Discreet Cosine Transform

A few thoughts on ruby, video and other things

Mounting a GlusterFS Mountpoint on Bootup in Ubuntu 14.04

I ran into an issue where despite the presence of the _netdev option in my fstab file, my GlusterFS mount point was failing to mount on system start.

There is a log in /var/log/glusterfs/ that pretty clearly showed it was trying to start before networking had started, as it kept failing on name resolution for whatever node I used in /etc/fstab.

Poking around a bit I found someone with the same problem on Server Fault. He had discovered a further log, in /var/log/upstart/ that showed an error message generated from the upstart script /etc/init/mounting-glusterfs.conf.

A little more reading showed that mounting-glusterfs.conf relies on another upstart job, /etc/init/wait-for-state.conf. This was the last clue I needed - as wait-for-state.conf expects a job name passed in as WAIT_FOR, not an event as mounting-glusterfs.conf tries to use.

So I changed WAIT_FOR in mounting-glusterfs.conf to networking instead of static-network-up and that resolved the issue.

Perhaps the wait-for-state.conf script has an updated version, as it seems some people are saying that mounting-glusterfs.conf works on other distros. So that might be a solution as well. But the change above is working for me on Ubuntu 14.04.

New Home for the Streamingmedia.com Advanced List

Streamingmedia.com decided to no longer continue the Advanced listserv they have maintained for the last decade. However, the members of the list decided to keep that community going. Come join if you are interested in discussing online video delivery and other topics related to video streaming!

Riak-client and :symbolize_keys

Also in the vein of “maybe this saves you a little time”, be careful of riak-client version 1.4.3 or less and setting

1
:symbolize_keys => true

in MultiJson. If you set it for all load opertions, you will break some of riak-client’s methods such as #keys, #buckets and even map-reduce.

For anyone of the “you found it, you fix it” mindset out there I did report the issue but admittedly did not create a pull request with a fix yet. I agree, shame on me!

Dartium’s Fineprint

I am just starting out with Dart and ran into a little hiccup using Dartium that took me longer than it should have to realize what was going on.

Buried on the download page is this line:

The Dartium binary expires after 12 weeks.

And what that line doesn’t detail, is that Chromium will still launch, however, it will just silenty no longer execute Dart scripts.

To fix this, download the latest version of Dartium from the download link above.

Understanding Riak-client’s WalkSpec

I had a little trouble initially grokking riak-client’s options for link walking. There is a wiki page on it on their github and some api docs on it, but I still couldn’t get my mind around the options. Here is a mini walkthrough that might be helpful if you are having the same problem.

Setup

Fire up IRB or your REPL of choice and load riak-client, then connect to your local riak server or wherever you want:

1
2
require 'riak'
client = Riak::Client.new(:protocol => "pbc", :http_backend => :Excon)

I created four objects to toy with, a through d. Note at least when using protocol buffers, riak-client doesn’t let you leave data blank, so I set that to the simplest json doc I could, all we need is the riak links:

1
2
3
4
5
6
7
8
9
10
bucket = client.bucket('test')
a = bucket.get_or_new('a.json')
b = bucket.get_or_new('b.json')
c = bucket.get_or_new('c.json')
d = bucket.get_or_new('d.json')

a.data = '{}'
b.data = '{}'
c.data = '{}'
d.data = '{}'

Then I chained them together with some simple links. This part is pretty easy, thanks to the #to_link method on riak-client’s RObject:

1
2
3
a.links << b.to_link('foo')
b.links << c.to_link('foo')
c.links << d.to_link('bar')

So you can get to c from a through the tag “foo” but you can’t get all the way to d, as that is tagged with “bar”.

Last part of setup, store all your objects:

1
2
3
4
a.store
b.store
c.store
d.store

Walking

There is a shortcut #walk method on RObject, though you can also call #walk from the client itself. I like the shortcut, and at least one use of it is reasonably clear:

1
a.walk('test','_',true)

returns:

1
=> [[#<Riak::RObject {test,b.json} [#<Riak::RContent [application/json; charset=UTF-8]:(4 bytes)>]>]]

So this means walk the links on RObject a, keeping all results to the bucket test, but following all tags (the _ character is the wildcard in riak’s scheme). The true means to return the results of this walk leg, which was one of the parts that was confusing for me at first but makes more sense later.

The returns make sense: it returns b but not c which is one more link away. It only walks the first link, in other words. But why are the results an array of an array? It makes it seem like the return could be two dimensional, but how to achieve that wasn’t real obvious at first.

Note that the api docs do make clear there is an alternate syntax here, using a hash:

1
a.walk(bucket: 'test', tag: '_', keep: true)

Which is nice because its more explicit.

See that syntax and reading the source for the #normalize method on WalkSpec had things making more sense to me. To walk two links out from your current node, just pass it to hashes, both of which will be turned into WalkSpecs:

1
2
a.walk({bucket: 'test', tag: '_', keep: true},{bucket: 'test', tag: '_', keep: true})
=> [[#<Riak::RObject {test,b.json} [#<Riak::RContent [application/json; charset=UTF-8]:(4 bytes)>]>], [#<Riak::RObject {test,c.json} [#<Riak::RContent [application/json; charset=UTF-8]:(4 bytes)>]>]]

Now the keep: true adds up, as does the double array. If the first walk had returned more than 1 result, then the second level of the walk would now be branching, and two dimensions of results would be returned. If we wanted to get to c from a but not return b:

1
2
a.walk({bucket: 'test', tag: '_', keep: false},{bucket: 'test', tag: '_', keep: true})
=> [[#<Riak::RObject {test,c.json} [#<Riak::RContent [application/json; charset=UTF-8]:(4 bytes)>]>]]

And if we want to get to d:

1
2
a.walk({bucket: 'test', tag: '_', keep: false},{bucket: 'test', tag: '_', keep: false},{bucket: 'test', tag: '_', keep: true})
=> [[#<Riak::RObject {test,d.json} [#<Riak::RContent [application/json; charset=UTF-8]:(4 bytes)>]>]]

But that only works because we have allowed for any tag, remember we tagged the link to d with bar so if we try:

1
2
a.walk({bucket: 'test', tag: '_', keep: false},{bucket: 'test', tag: '_', keep: false},{bucket: 'test', tag: 'foo', keep: true})
=> [[]]

I might write a larger test script and test data set to really play with multiple levels of walking and those nested results, but overall the logic of #walk is a lot clearer to me now.

Marshal Ruby Objects to Riak and Back

Thought I would share a little bit of work I have been messing around with to marshal ruby objects to the Riak NoSQL store and back. There is Ripple, a library for exactly this purpose, but it implements the ActiveRecord model and that is not exactly what I wanted.

riak-client

Riak client supports automatic serialization based on what you set as an object’s content-type. So for example, the follow code would automatically store this Ruby hash as JSON in Riak:

1
2
3
4
5
6
client = Riak::Client.new(:protocol => "pbc")
bucket  = client.bucket("my_bucket")
new_one = Riak::RObject.new(bucket, "hash_object.json")
new_one.content_type = "application/json"
new_one.data = one
new_one.store

The difference here is in the use of #data and not #raw_data. There is a way to define your own serializers for different content-types, but that seems to be undocumented by the riak-client team for now. By default it supports both YAML and JSON.

That will work fine for the typical JSON data types such as Hash and Array. But if you want to store a more complex object, and get it back, you will need a little more.

Using the JSON Gem

riak-client uses multi_json as you might expect. So, if you have the JSON gem installed or loaded, you can use their method of storing a ruby object and marshaling it back from JSON.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class ComplexObject
  attr_accessor :place,:time,:characters

  def initialize(stuff={})
    @place = stuff[:place]
    @time = stuff[:time]
    @characters = stuff[:characters]

  end

  #we have to define our own #to_json and self.json_create methods
  def to_json(*a)
    {'json_class'=>self.class.name,'place'=>@place,'time'=>@time,'characters'=>@characters}.to_json(*a)
  end

  def self.json_create(data)
    new(:place=>data['place'],:time=>data['time'],:characters=>data['characters'])
  end
end


one = ComplexObject.new(:place=>'woods',:time=>'tomorrow',:characters=>'burt')


client = Riak::Client.new(:protocol => "pbc")
bucket = client.bucket("my_bucket")


#in case I have other JSON libraries loaded that MultiJson would favor
MultiJson.use :json_gem

#this is a key, as it does not default to true 
#and without it the JSON gem won't call json_create
MultiJson.load_options = {:create_additions=>true}


new_one = Riak::RObject.new(bucket, "complex_object.json")
new_one.content_type = "application/json"
new_one.data = one
new_one.store

#viola, you get back a new object, not just a Hash
puts client['my_bucket']['complex_object.json'].data

Using Oj

It appears Oj has even more innate support for serializing and deserializing ruby objects to JSON, but I can’t say I am deeply familiar with Oj yet so take that with a grain of salt.

Here is the example above using Oj:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
class ComplexObject
  attr_accessor :place,:time,:characters

  def initialize(stuff={})
    @place = stuff[:place]
    @time = stuff[:time]
    @characters = stuff[:characters]

  end

  #with oj, no need for our own serialize and deserialize methods

end

one = ComplexObject.new(:place=>'woods',:time=>'tomorrow',:characters=>'burt')

client = Riak::Client.new(:protocol => "pbc")
bucket = client.bucket("my_bucket")

#MultiJson would default to oj if present, just being explicit here
MultiJson.use = :oj

#we want oj's object mode to do its automatic serialization
MultiJson.load_options = {:mode=>:object}
MultiJson.dump_options = {:mode=>:object}

#Store a ruby hash as json in riak

new_one = Riak::RObject.new(bucket, "complex_object.json")
new_one.content_type = "application/json"
new_one.data = one
new_one.store

#viola, again, the full ruby object
puts client['my_bucket']['complex_object.json'].data

Planescape

Planescape was my favorite Dungeons and Dragons setting of all-time.

I think admittedly, it was a little more “advanced” than say the Forgotten Realms. What I mean is, if you were playing D&D for the first time with someone new, sitting down and explaining Planescape was perhaps a bit harder, and it meant perhaps a bit more to people that had already played for years with the planes as a backdrop to magical adventures.

But, to contradict myself a bit, the flip side of that is Planescape definitely appeals to a number of core fantasy desires and concepts: hopping into a glowing portal and ending up in a different place is a quintessential fantasy moment. And even someone who has never read a fantasy book or played D&D will probably have a concept of heaven and hell, and know a bit about Greek myth for example - in these ways I think its possible Planescape would be even more digestible then say the Forgotten Realms to some new users.

Either way, the setting had an incredibly strong sense of place for me. That was due to both great writing and fantastic illustrations, in addition to the great core concepts. The illustrations by Tony DiTerlizzi remain some of my favorite to this day.

Also a little contrary to my previous posts, I think Planescape accomplished some of the goals I was bemoaning previously, that Wizards of the Coast brought to Dungeons and Dragons - the push towards fantasy settings that more closely resemble video games like World of Warcraft. It would seem WotC agrees with that, since the Tiefling, first introduced in Planescape, has become a more core part of the game and other settings (and indeed, strikes me as a character that would be right at home in a World of Warcraft setting).

I am surprised it was not more popular at the time of its release, though judging popularity is a relative thing. However, I do think it was far ahead of its time. It was first published in 1994, long before World of Warcraft or the current popularity of concepts like steampunk. However, a (very cursory) ready of current D&D forums online shows its not the most popular setting even now.

That is ok, I am not interested in a popularity contest, I just think the setting deserves more praise. My ideal game is a AD&D 2nd Edition romp set in Planescape.

Object-Oriented Story Telling

Being in both computer programming and role playing (as I would guess many old D&D players find themselves - geek begets geek after all), I have often thought about the concepts of relational data, object-orientation, and story telling.

In relational data, one record relates to another via some common data. There are a million glib examples, like looking up the weather for your hometown via the zip code (a more detailed way to say it might be you can find approximate latitude and longitude via the zip code, and find weather via lat and long - those that have worked with relational data before can almost picture the SQL statement in their minds right now).

Developers then spend a lot of time modeling real-world things and processes into software objects - single units of both functionality and data, with the very common trite examples being something like you have an ”order” object that contains many ”item” objects. Often, you even take it a step further and map those objects onto relational data - orders and items become records in a relational database. Common strategies for doing this exist, such as object-relational mappers (ORMs) and diagramming techniques, while not the fad at the moment, exist, such as relational database diagrams and object models in UML. A common long exists for these types of relationships, that talk about things like the definition of a given specific object (an “is a” relationship), the relationship between different objects (“has one”, “has many”, “has many through”, etc.) and how to relate one object to another. So a zip code has a general latitude and longitude, a town has a zip code, and a bunch of records that describe temperature, barometric pressure and wind speed all have a latitude and longitude to describe where they were measured or are predicted to occur. Viola, I input the zip code, I get out a weather forecast.

Story telling, maybe particularly interactive story telling, strikes me as having some similar ideas. A character has friends and enemies, possess items of power or items other people desire, and resides or adventures in a location. That same location houses other characters, and has traits of its own: geography, weather, and in the case of fantasy perhaps magical properties. It also has history: events that happened there in the past, and those evens in turn involved other characters (or your character). You can even think of useful queries you might perform against that data - say as a dungeon master, or just as author trying to keep all your notes straight. What are all the events that happened here, at a specific location? When was a specific character in this location? If the character possessed that highly valued item of power at the time (the ring!), then logic would apply that the item of power was in that location at the same time (though magic, of course, always jeopardizes logic such as this!).

No, I have not put this grand plan into action yet, but its on my mind per my previously stated goal of finding where my current life still intersects Dungeons and Dragons.

A Sense of Place

One of the things I always particularly loved about D&D, and fantasy in-general, is the sense of another place, the feeling of being there.

However, I think that sometimes who are not fantasy fans might assume that the definition of a “fantastic place” must mean something filled with a lot of danger, or even a place filled with truly fantastical things. I think video games have indeed helped perpetuate this - a “good level” in a video game often is exactly that: filled with danger and fantastical things. Floating islands, erupting volcanoes, enemies around every turn, etc.

That is not what I necessarily enjoyed though. Think instead of the early scenes in Peter Jackson’s interpretation of The Fellowship of the Ring - all those scenes in the Shire. What a wonderful job Mr. Jackson did of visually creating a place that fit the voice-over describing the peacefulness and contentedness of the hobbits. Isn’t that Shire a place you want to be? A place you want to spend time?

There were many such places in AD&D 2nd Edition - the Forgotten Realms, of that edition, had many such places. Yes, it had places filled with danger and intrigue too, but it was very much the type of fantasy that makes it easy to imagine a beautiful forest glen, a high mountain pass in spring, a trading port on the sea on a clear blue day. The more fantastical places had some sense of that too - Sigil, in the Placescape setting, always seemed like a great place to be to me. Yes, a little more “danger around every corner” then perhaps some of the Forgotten Realms, but still, it was easy to imagine an Inn or other gathering in that metropolitan city that was just as much a source of relaxation to its urban inhabitants.

I guess what I really mean by A Sense of Place is anywhere that you can experience with more senses than just your eyes - of course, something that AD&D excelled at because you imagined so much of it (versus say a movie, which I think has a bigger challenge to make you feel that place with more than just your sense of vision). If you can almost feel the breeze on your face, or smell the food vendors, and you get an actual tingle that says “I want to be there”, that is what I mean.

While I do think that is in a strange way harder in a movie, where best efforts can still generate an image that to you feels like looking at a picture, not being there, there are some very strong examples of this in movies as well. Ridley Scott particularly excels at it, in my opinion, and movies like Blade Runner really put you there.

Interestingly, supporting the idea that calling on the viewers imagination makes this more effective, I think some anime movies are perhaps the absolutely strongest film-examples of this. The Place Promised in Our Early Days, Wings of Honneamise, and most especially Spirited Away are some of my favorite examples. If you can’t sense the feeling of being in the bath house in Spirited Away - smell, taste, sight and everything - I might go so far as to say your crazy.

Which is all to say, that at least for me, good fantasy and good role playing are not necessarily about constant danger, and don’t need to be so fantastical as to be unrecognizable. I will go so far as to say video games (which, don’t get me wrong, I do love) are a poor model in some ways for table-top role playing. The pacing, the intent, and the way they confront your senses are generally different. There are a handful of examples of video games that are slower paced and have a true sense of exploration over constant jeopardy, but I doubt those were the games you thought of when I first said “video games”.