Marian's Blog

3D Printed Mechanical Digital Clock

Sun, 06 Apr 2025 00:00:00 +0000

This post is about building a 3D printed, mechanical digital clock, made out of seven-segment displays. The project was inspired by a video by Lukas Deem, who built a clock with seven-segment displays where the segments slide in and out when they change their state. In his design, each segment is controlled by a separate servo. I was also inspired by various projects from the Karakuri channel, that use camshafts to drive seven-segment displays entirely mechanically.

I wanted to combine the sliding in and out aesthetic of the white segments from the former with the mechanical approach from the latter.

Cam designs

To display ten different numbers, each of the seven segments needs to go through a sequence of ten stops, where the segment is either active or inactive. Here is an early test where I printed a camshaft to push a follower back and forth. It already has ten stops and this prototype confirmed that my code was working to generate 3D cam shapes based on a sequence of segment states.

Here are some early iterations of digits I printed:

It has two camshafts with seven cams, so that each segment can be pushed out by one cam and pulled back in by the other. At the bottom, the camshafts are connected by a pair of gears so that they rotate at the same time.

A lot of iteration was done to make the digits run reliably, without too much resistance and without getting jammed. Another big improvement was putting the pushing and pulling cam on the same camshaft, reducing the overall complexity and part count. This is the design I ended up with:

Each of the seven pushers has two pins, one above and one below, which engages with a cam to push it in one or the other direction. This could also be achieved with a single cam and a spring, but I wanted to avoid tensioning the parts.

You can see that the side panels ended up being quite thick. When the camshaft pushes against the pins, it's important that the sliders move back and forth, instead of the walls bending and the sliders moving left and right. With the final design, everything should move smoothly right after printing but adding silicone spray helps a lot with 3D printed mechanical projects.

Connecting four digits

Having designed one digit, the next step is to connect multiple digits. For this, I'm using a mechanism called Geneva drive, which will advance the tens digit of minutes by one when the ones digit of minutes goes from 9 to 0. This is what it looks like:

Note that the ones and tens digits have different camshafts, the ones digit cycles through the numbers 0 to 9, while the tens only goes up to 5 and it goes back to zero after minute 59.

Another interesting constraint is that both digits need to advance by one number while the pin of the Geneva drive is engaged. That means that the ones camshaft advances by 360/10 degrees, since it has 10 stops and the tens cam advances by 360/6 degrees, since it has 6 stops. This requirement fully determines the position of the pin and the radii of the two wheels of the Geneva drive. In a standard Geneva drive, the pin enters and leaves the notch straight on, which is not the case in my design since it would dictate the distance and radii of the wheels.

The hours digits also have unique camshafts, although there is a special case with the ones digit of hours. To display 24 hours, the digit goes through 0 to 9, then through 0 to 9 again and then through 0 to 3. After that, at midnight, it transitions from 3 back to 0. I considered different ways of implementing this mechanically. It would be cool to have 10 digits and somehow skip from 3 to 0 at every third run. Eventually I decided to just make a camshaft with 24 stops, which is not as elegant but solves the problem. As a result of that, the ones digit of hours is bigger than the other digits.

The tens digit of hours has a similar problem, it only needs to show 0, 1 and 2. This would mean that the Geneva drive needs to move 120° when it engages, which is too much. For this reason, the camshaft of the tens digit of hours stops at these three numbers twice, for 6 total stops.

These are the four camshafts in the clock:

Here is the final design with all three Geneva drives engaging simultaneously when the clock rolls over at 23:59:

Note that the large Geneva drive wheel has three pins placed so that they engage after 9, 19 and 23 hours.

Motorization

The next step was to motorize the clock. For it to function as a real clock and display the time, it needs to advance once a minute. I decided to use a stepper motor for this as it allows for precise angular movement.

Another advantage of using a stepper motor for mechanical art projects is that it is completely silent. When the clock advances to the next minute, all you can hear is a little mechanical rattling. The motor itself makes no noise. You might think that a stepper motor is entirely sufficient to accurately drive the clock, but in practice this is not the case. When turned on, the stepper motor will consume a fixed amount of power, even when not moving. To save power, I turn it off between each minute advancement. The motor holds its position while powered but moves with less resistance when powered off. Sometimes this leads to small errors, which accumulate over time.

I printed a mounting bracket for the motor and a gear to transmit the power. I use a TMC2209 stepper motor driver and a Seeeduino XIAO since it is compact and has USB-C. The project also has a voltage regulator so that the 12V supply for the motors can be used to power the Arduino and three buttons to set the time.

Rotary encoder

To avoid accumulating errors while the stepper motor is turned off, I added a rotary encoder to the ones of minutes axis. The gear now has ten protrusions that interrupt an IR photo interruptor.

The photo interruptor consists of an IR LED and a photo sensor. When a part moves into the interruptor, the sensor can no longer "see" the LED. Note that the protrusion for the zero digit is a bit thinner. By measuring how long the sensor is interrupted, the Arduino can tell if the current digit is zero.

Here is the mechanism in action:

I also updated the design a bit to allow a single front panel. The clock is made up of four digits, the "colon" separator panel in the middle and top and bottom parts to hold it all together. These segments were visible from the front, but this new design avoids all the gaps by using a single decorative front panel.

Keeping time and practicality

The clock has a power supply that plugs into a wall outlet. As long as it has power, it will accurately advance its time once a minute. It doesn't produce much noise, just a subtle mechanical rumble once every minute. But every time the clock is turned on, the correct time needs to be set. For this purpose, the clock has two buttons to go forward and backward.

I've considered adding a real-time clock module, which uses a tiny coin battery to keep the time even when the device is turned off. For it to work, the Arduino would have to write the current RTC time to its EEPROM every time it moves. This information is retained when the device loses power. When it's turned back on, it would then have to read the current time from the RTC and the current clock state from the EEPROM and fast forward the clock by the difference between those two.

Another consideration is that the EEPROM in the Arduino has a limited number of write cycles. I don't know how this will play out in practice, but if the device is limited to 100,000 writes, it could only operate for a few months.

I haven't done this yet, but it's the one remaining feature that I have in mind for this project. Since I've just now written the blog post, the project is complete and this probably won't happen. But I wanted to mention it so that you know I've thought about this problem!

Conclusion

One common piece of feedback I've received relates to the contrast of the segments. I've specifically chosen this style where the segments slide in and become invisible. Yes, the contrast is low but in practice it's not really a problem. It's tricky to catch this on photos but in real life, you can easily tell the time. Also, it's more of an art project than a practical clock. There are ways to make the segments pivot out of view, but it just doesn't look as nice.

If you'd like to make your own version of this clock, you can get the files for free on Printables. You can also just print a single digit. If you're interested in printing mechanical seven segment displays, also check out this design (not by me), which has segments sliding in and out of view for higher contrast.

Designing a Lego orrery

Sat, 01 Jun 2024 00:00:00 +0000

I've always been a fan of the Lego Technic series, especially those models that have gears and cranks and moving parts. But it seems that Lego is shifting the focus of the Technic series away from functional models, so I had to take matters into my own hands. I think an orrery is the perfect project to build out of Lego Technic parts as it makes for a cool display set and is functional at the same time.

Introduction

An orrery is a scientific instrument that models the motion of the celestial bodies. These are typically the orbital period and the period of the rotation around the axis of a body. For example, the time it takes the moon to travel around the earth once is ~27 times the time it takes the earth to rotate around it's own axis (a day). This relation is accurately modelled in an orrery. However, the orrery does not model distances or the relative sizes of the celestial bodies.

I was very excited when I saw this Earth, Moon and Sun Orrery design by JK Brickworks in 2016:

This inspired me to build my own orrery with the same spheres for the sun, earth and moon, but my own custom designed mechanism:

While working on this, I experimented with 3D printing custom fixtures to put Lego gears in, but eventually moved on to building an orrery out of Lego parts only. This orrery is special in the sense that it was designed by me, but it doesn't have any features beyond what the design from JK Brickworks has to offer. It models the orbit of the earh around the sun, the orbit of the moon around the earth, and the spin of the sun, earth and moon.

The moon is tidally locked to the earth, meaning it rotates once per orbit. One half of the moon always faces the earth and the other always points away from it. (This also means that if you're on the surface of the moon, the earth always stays in the same spot in the sky.) The tidal locking means that the rotation of the moon easy to implement in an orrery, since the moon will move correctly as soon as it is attached to an arm that goes around the earth at the correct orbital period. You only model the orbital period and get the rotational period for free.

To improve upon my first model, I wanted to add two more features: The axial tilt of the earth and the inclination of the orbit of the moon.

Axial tilt

The rotational axis of the earth is tilted 23.5° relative to its orbital plane. For the orrery, this means that the rotational axis of the earth is not pointing straight up. To make matters worse, the axis always points in the same direction. It does not rotate relative to the stars or the table that the orrery stands on, but it has to rotate relative to the main arm that holds the earth. So not only does the earth itself rotate, the part that holds the earth also needs to rotate to cancel out the rotation of the arm.

Modelling this feature allows us to use the orrery to explain the origin of the seasons.

To construct the actual tilted holder for the earth sphere, I used an angled connector with a 22.5° angle, which is only one degree off the correct angle.

Inclination of the orbit of the moon

The second pehonomenon I wanted to model in my orrery is the fact that the plane of the moon's orbit is inclined relative to the plane of the earth's orbit. The real inclination angle is 5.15°, which you would barely notice if it was modelled accurately. But the inclination of the moon's orbit plays in important role in explaining when eclipses occur, so I think it is useful to model an exaggerated orbital inclination.

In the orrery, we need to make the moon go up and down along its journey around the earth. My mechanical solution for this problem is inspired by this video. The moon arm has a wheel that rides on a tilted circular gear rack. A parallelogram linkage keeps the moon vertically aligned. Just like the attachment point for the earth, this assembly rests relative to the stars, meaning it needs to rotate relative to the main arm.

In summary, there are four parts on the earth arm that need to rotate at the correct rate:

the earth,
the base for the earth (to ensure the correct direction of the axial tilt),
the moon,
the inclination ring for the moon.

In theory, the earth base and the moon inclination ring could be a single part (both should rotate at the same rate). But making a tilted ring out of Lego parts is difficult enough and I didn't manage to do it in a way where the moon "rides" on it while it is driven from the outside. Thus, the solution to this is to make the moon inclination ring a separate assembly.

Being able to move the tilted ring independently allows us to implement an additional astrodynamic phenomenon, the precession of the nodes of the orbit of the moon. It turns out that the direction in which the upper and lower parts of the orbit are pointing isn't fixed after all. They slowly rotate around the moon, once every 18.6 years.

Concentric axes

One problem specific to building an orrery in Lego is achieving concentric rotation. In a traditional orrery, you can have an unlimited number of parts rotating around the same axis just by placing tubes of increasing size inside each other. But these tubes don't exist as Lego pieces. For the earth-moon system, I need four assemblies rotating independently around the same axis.

Here is how I achieved it for the earth-moon subassembly: The first axis (1) is a regular Lego Technic axle. It drives the rotation of the earth. The second axis (2) is built using a Lego Technic turntable. It is responsible for the orbit of the moon. The third axis (3) is a circular gear rack. It fulfills the same role as a turntable, just with an inner radius large enough to fit another turntable inside it. This part is the tilted ring that is needed to model the orbital inclination of the moon. Now there is only one axis missing, which is the rotating holder for the earth (4). Remember, it needs to rotate relative to the main arm of the orrery so that the rotational axis of the earth points at a constant direction. This is done with a second turntable stacked on top of the first one.

To drive it, I'm using this trick: The Differential Gear Casing has a round hole and gears at both ends. That means an axle can be placed inside it and rotate independently. This can be used to transmit torque by connecting gears at both ends. This serves the purpose of the aforementioned tubes that don't exist as Lego parts, with the exception of this gear casing. The gear casing drives some gears on the first turntable, which in turn drive the second turntable.

Gear sequences

The earth-moon subassembly has four axes that need to rotate at the correct period, namely the earth, the moon, the earth's base and the moon's inclination ring. The entire orrery has two more: the rotation of the sun around its own axis and the rotation of the main arm, which makes the earth travel around the sun in 365.25 days.

The sun is not a solid body and spins at different speeds depending on the latitude. For orrery design, that means that any rotational period between 24 and 38 days can be argued to be correct, since there is a latitude of the sun where the surface rotates at this period.

This gives rise to the problem of finding a gear sequence for a given transmission ratio. In my orrery, one rotation of the crank corresponds to one day. So to get the main arm to rotate once every 365.25 days, it needs to be connected to the crank using a gear sequence with a transmission ratio that is as close as possible to 365.25.

The gear ratio of a gear sequence is the product of the fractions of tooth numbers for each pair of gears in the sequence.

To find a sequence of Lego gears, there are some additional constraints. Most importantly, we are limited by the gears that Lego actually makes:

Notice the worm gear in the above image. It's special because it behaves like a one tooth gear, i.e. paired with a 24 tooth gear, it will create a transmission ratio of 1:24. But it can only be the driving gear in a pairing, not the driven gear.

Suppose we want to achieve a transmission ratio of 17:23. This could be done easily with a pairing of a 17-teeth and a 23-teeth gear in a traditional orrery. But these teeth numbers aren't available as Lego gears, so we have to find another way.

I made a tool to find Lego gear sequences for a given transmission ratio. For our example of a 17/23 ratio, the app finds plenty of solutions with slightly different ratios, for example a ratio of 20/27, which is only 0.22% off from the target ratio:

The problem of finding a gear sequence for a given transmission ratio is connected to some interesting ideas from number theory. It is closely related to prime factor decomposition. In fact, an exact solution is only possible if all the prime factors in the target ratio are also available as prime factors in the available gears.

The tool will start from the target fraction (or slightly different fractions when searching for approximate solutions) and go through a list of expansions of that fraction that are possible with the prime factors given by the available gears. For each fraction, it will calculate all sequences of gears that have the numerator or denominator of that fraction as their product.

Given such a pair of sequences of driver gears and follower gears, the app still needs to decide which driver gear is paired with which follower gear. This is important in the context of Lego gears because some pairings are more practical than others in a Lego project. In particular, if a pair of gears has a tooth sum that is divisible by 16, the distance between the axles is a whole number of Lego units. This combination is the easiest to build.

Here is an example of pairing a 24 and 40 tooth gear. The tooth sum is 64 = 4 * 16, which tells us that the gears will be four Lego units apart.

If the tooth sum is divisble by 8, the distance can be achieved by placing the axle at a half unit offset, which is possible with various building techniques:

The app has a "Fit gears" tab which will show you possible ways of connecting any given pair of gears. For example, this is what I call a "2D connection". These gears fit together if you offset them 1 unit in one direction and 2 units in the second direction, since their distance is quite close to sqrt(1² + 2²):

In addition, there are also gears that Lego calls "bevel gears", they have n * 8 + 4 teeth and can be connected perpendicularly, allowing you to change the plane of the meachnism.

So for any pair of a driver and follower gear, there are potential ways to connect them, but some combinations are easier to build and some are harder. My app assigns a "cost" to each possible pairing of driver and follower gear based on how they can be connected. If the teeth numbers add up to a multiple of 16, they are the easiest to connect, meaning the cost is the lowest. Once the tool has a set of driver gears and follower gears, it uses the Hungarian algorithm to generate a matching with minimal cost.

Here is an example of a result when searching for a sequence with a ratio of π:

There is one more interesting mathematical curiosity I want to mention. Lego's differential gears have the property that the differential casing rotates at a rate that is the mean of the rates of the two connected axles. For example, if one axle rotates once per second and the other axle three times per second, the casing will rotates twice per second ((1 + 3) / 2). We can exploit this to achieve prime factors that are otherwise impossible with Lego gears.

For example, there are no Lego gears with a tooth number that is a multiple of 13, so transmission ratios that include 13 as a prime factor are not possible with the standard approach. To achieve a transmission ratio of 13, the gear ratio calculator gives us this sequence:

The inputs axles spin at a rate of 1 and 25 and the resulting rate is (1 + 25) / 2 = 13. This how this sequence could be built:

In this example, the blue axle spins 13 times as fast as the red axle.

This brings us to the end of the digression about gear sequences. The app is available here if you want to try it out. The Typescript source code is also available.

Here are all the gears in the final orrery design:

These are the rotational and orbital periods in the orrery, comparing them to the real-world values:

The mechanism was mostly designed using physical parts and my gear sequence app. This is a work in progress photo:

One of the main challenges during the design of the orrery, besides making it work at all, was to get the orrery to run smoothly. The final design has 70 gears and each additional gear adds a little bit of friction. Another source for resistance is the weight of the earth as it spins relatively fast and thus contributes a lot to the torque needed to run the orrery. I put a lot of effort into making the orrery run smooth. But the mechanism can easily get stuck, for example if any moving part is assembled too tight or if the orrery gathers dust. During assembly, it is important to make sure that every moving part has a some slack.

One way to work around this problem is to apply a little bit of silicone spray. This will make the orrery run effortlessly without any problem. But arguably a Lego set should be judged by how well it works on its own and purists will say this is like gluing parts together.

Designing the foot

I took this photo when I first managed to get the entire mechanism to work. This contains the first version of a foot I designed, which has a Lego motor to drive the orrery.

It turns out that this motor is really loud when it runs and it ruins the experience of using the orrery. I envision it as a piece that you can use to explain and demonstrate astrodynamic phenomena to someone. But this doesn't work when the motor noise makes it impossible to have a conversation, so I decided to remove the motor.

Designing the foot turned out quite tricky, it's an artistic challenge more than a technical one. I went for a 12-segment design, which is a nod to the twelve months and has a tradition in orrery design.

I designed most of the foot digitally using Bricklink Studio. This part lends itself more to digital design since it doesn't move. Due to the 12-fold symmetry, testing any idea requires a large number of parts.

Here are some iterations on the foot design:

You can see that each design has a little lever to disengage the driving mechanism from the main arm. This allows you to move the arm freely, which is quite useful in practice.

Here is a video of the final orrery design in action:

Instructions

I used Bricklink Studio to make a digital model of my orrery. During the design process, I switched back and forth between buildig with physical parts and designing digitally.

I also used this program to create digital building instructions. My takeaway from this is that it works really well for small models, but it has serious usability problems on larger models. When you make changes to the model or change an early step in the instructions, it will often break the layout of the instructions in many places so that most of it needs to be redone. This was a lot of tedious work, but I ended up with a 264 page, 436 step instructions PDF.

I'm selling the digital instructions for 20€ on Rebrickable. On Rebrickable, you get a parts list and some really useful tools to help aquiring the parts (even without buying the instructions). For example, you can tell it which Lego sets and parts you own and based on that it will know which of the required parts of the orrery you already have. It also helps you with buying the parts you don't have. Rebrickable estimates that the total cost of the required parts is 260€. Together with the cost of the instructions, this makes the whole project quite expensive for potential buyers. I ended up selling a few instructions, but ultimately this project wasn't a commercial success. In hindsight, it might have been a better idea to give the instructions away for free.

To promote the listing on Rebrickable, I made an animation that shows the movement of the gears in the orrery. I used Mecabricks to convert the Bricklink Studio design into a scene file that contains 3D meshes of all the parts. This could then be opened in Blender, where I created all the animations and the camera movement. Just like making the instructions, this is realtively straight forward, but it became quite complex due to the large number of parts in the design. Here is the resulting animation (at 45 seconds, the structural parts fade out to reveal the mechanism):

Lower partcount design

The above design aims to make the best possible Lego orrery with few compromises. This is at the expense of having a high part count and some parts that are hard to get. I made a modified design with a lower part count by redesigning the foot.

The foot in this version is much lighter, has less ornaments and uses Technic parts instead of traditional bricks. It has one crank instead of two. To make it easier to get the parts for this orrery, I changed the design so that it uses as many parts as possible from Lego's Rough Terrain Crane. I chose this set because it was well reviewed, had a low cost per part, and many people own it. Unfortunately, Lego has now stopped selling it.

Many parts in my first orrery design use a "traditional" color scheme where gears, pins and axles are mostly gray and black. This color scheme was used by Lego in the past and is often preferred by fans. In the new design, I use the same colors that Lego uses in its current products, which means the parts are easier to get and more of them can be taken from the red crane. The new, more colorful palette makes the building instructions easier to follow, but the colors can look out of place in the orrery.

Other orreries

I released the instructions for my orrery in September 2021. A few months later in 2022, CaDA released an orrery designed by JK Brickworks. This is the designer behind the "original" Lego Orrery from 2016, which got me interested in this hobby in the first place.

The feature set of this orrery is similar to mine, with the exception that it doesn't model the inclination of the orbit of the moon. As you can see, omitting this feature lowers the complexity considerably and allows for a much simpler and more reliable design. The lower part count makes this orrery more accessible to a wider audience. I'm very impressed with this design and I think it's the best product to get for anyone interested in building an orrery.

Two years later in 2024, Lego released their own orrey design. I was quite excited at the prospect of an official Lego orrery, but I was disappointed by the execution.

This orrery has the same feature set as the one from CaDA, it has the earth's axial tilt, the moon's orbit but not its inclination. Compared to the CaDA design, it has less parts but is more expensive. I think Lego were too aggressive in reducing the part count. The orrery is very fragile and wobbly. For the sun and earth, they went with custom hemisphere pieces. I like the idea, as it allows the bodies to be perfectly spherical. But the parts have very prominent connection holes, which are unneccesary and ruin the look. The earth has the continents printed on it, which is a nice idea but the prints were poorly aligned in my model.

All in all, I would recommend the CaDA orrery over this one. But I'm excited to see that building block orreries are reaching a bigger audience and I hope that companies like CaDA and Lego will offer more elaborate designs in the future.

Outlook

This is a mockup of an idea to model all the planets in the solar system without vertical stacking (well, except for the inner planets, but it's mostly a flat design). The four outer planets would move on four tracks of chains that are held in place by the vertical poles (the chains are not shown in the render). The chain would be driven by a gear attached to the base and the planet would be attached to one of the chain link pieces. Note that in this setup, the planets only travel around the sun, but don't spin around their own axes.

This concept is inspired by a design for a clock from Akiyuki. It's hard to tell if the chain could handle the weight of a brick-built planet. Either way this build would be crazy in terms of dimensions, part count and cost. At the moment, I'm not planning to continue working on this concept.

If you remove the constraint of no vertical stacking, the problem becomes only slightly easier, but possible as demonstrated by Chris Orchard and Brent Waller in their Lego Ideas campaign.

My next project is to build an orrery using laser cut acrylic sheets instead of Lego parts. This removes a lot of constraints on the gears and parts that can be used, but has the added difficulty that I need to manufacture these parts myself.

Generating an infinite world with the Wave Function Collapse algorithm

Tue, 20 Jun 2023 00:00:00 +0000

This article describes how I generate an infinite city using the Wave Function Collapse algorithm in a way that is fast, deterministic, parallelizable and reliable. It's a follow-up to my 2019 article on adapting the WFC algorithm to generate an infinite world. The new approach presented in this article removes the limitations of my original implementation. I first mentioned these ideas in this Twitter thread.

Objective

The goal is to procedurally generate a 3D environment by placing human designed blocks on a 3D grid. The blocks need to be placed in accordance with given adjacency constraints. For each of the 6 sides of a block, some information about the face and its symmetry is used to generate a list of possible neighbors.

This is different from the original formulation of the WFC algorithm, where the possible blocks, their adjacency rules and their spawn probabilities are extracted automatically from an example texture.

With the improvements described here, the generation method is robust enough to be shipped in a commercial game. For this, it needs to be reliable, fast and allow for artistic control over the result.

Wave Function Collapse

This article is aimed at readers who already know how the WFC algorithm works, but here is a brief recap. I'm skipping the part where blocks are extracted from an example texture and only use the part where we generate a new "texture".

(Gif by Maxim Gumin on Github)

The algorithm starts with an array of slots (the "wave function"), where nothing is decided. Each slot has a list of possible blocks (or "modules") that can be placed there and in the starting state, each list contains all modules. The algorithm will then do collapse steps and a constraint propagation steps until the map is fully collapsed or until it has reached a dead end.

Collapse step

We pick the slot with the lowest entropy and collapse it. That means we pick one of the modules from that slot's list of possible modules and set it as the selected module for this slot. Intuitively, the "slot with the lowest entropy" is the one with the least amount of choice. If all modules have the same spawn probability, the slot with the fewest possible modules is the one with the lowest entropy.

Constraint propagation

Collapsing a slot effectively shrinks the list of possible modules of that slot to 1. The constraint propagation step propagates this information through the map by removing modules from the respective lists of other slots that have been ruled out by the latest collapse step. The constraint propagation step of the WFC algorithm is the most compute intensive part.

End

The algorithm terminates when all slots are collapsed, which means success, or when the list of possible modules for any slot is empty. In that case the procedure has failed and one could backtrack or start over.

The original approach and its limitations

(skip this section if you just want to know the solution)

WFC is usually applied to finite maps that can be stored in an array. In my original post, I described why I thought it would be impractical to do a chunk-based WFC implementation. I had not figured out how to avoid the problem that constraints need to be propagated across chunk boundaries. Instead, I stored the map in a dictionary, where new slots would be allocated when they were generated or when they were touched by constraint propagation. That means, even to geneate a small area of the map, a large cloud of slots around that area would be allocated since constraint propagation could "bounce back" into the area we're interested in.

Problems of that approach include:

Non-determinism: The result of the generation depends on the order in which parts of the map are generated (and thus on the path the player takes).
Memory leak: We can't release the memory used to generate a part of the world when the player leaves since we don't know at what point distant slots no longer have an effect on local world generation.
Reliability: The longer you walk around, the higher the chance becomes that the WFC algorithm runs into a dead end and is unable to continue generating the map.
Single threaded: Since there are no chunks, all operations on the map datastructure need to be sequential and can't run in multiple threads. In practice, the map height had to be limited so that the map generation was fast enough.

My implementation of the old approach is still available on Github and a playable demo is on itch.io. If you want to implement your own WFC algorithm, you shouldn't do it like that though!

In the next part, I'll describe my solution to all of the aforementioned problems.

Chunk-based WFC

The idea is to start with a simple, pre-generated, tiling map and generate "fitting" replacements at runtime. However, we do this at an offset so that the seam at the chunk borders (which would otherwise look the same for each chunk) is replaced. In this section, I'll explain in detail what that means.

This solution is a refinement of ideas proposed by Paul Merrel and BorisTheBrave.

We start by generating a simple, finite, tiling, 8x8x8 map:

This is done offline. We use a small subset of the available modules to make it as simple as possible. The map is tiling in the sense that the boundaries on opposite sides match, so copies of this map could be placed next to each other seamlessly. Doing that would look like this:

Generating the tiling map is done by "wrapping around" the constraint propagation at the map boundary. In a finite map, when we propagate a constraint to a slot outside the map, we discard that information. In a tiling map, the slot on the opposing map boundary is treated as if it was a neighbor.

Next, we pre-generate a set of replacements for our starting map.

We use the boundary of the starting map as a boundary constraint to generate these replacements. For any slots on the boundary of these new maps, we only allow modules with a matching profile for those sides that face the map boundary. This means that we can "swap out" our starting map with any of the pre-generated patches without getting mismatched blocks at the boundary.

Now we can randomly choose from our collection of pre-generated patches at runtime and we have a simple chunk-based infinite world generator:

Note that we're not doing any WFC generation at runtime yet, we're just placing pre-generated 8x8x8 block patches. Since all these patches have a matching boundary, we can spot this chunk boundary as an unnatural pattern in the generated world.

Now for the important part: At runtime, we generate an 8x8x8 replacement map for each chunk, but we do it at a 4 block offset in both horizontal directions. The starting point for each chunks's generation is made up of the four pre-generated patches that touch it. The replacement map we generate at runtime has a boundary constraint to "fit in", just like our pre-generated patches. However, due to the offset, the boundary that is shared between all pre-generated patches is replaced at runtime and the area that is different in every pre-generated patch remains unchanged during the runtime generation. (This is needed so that neighbor chunks can be generated independently from each other.) If this replacement map fails to generate, we just keep the blocks from the starting patches.

Here is the result of that:

Notice how the chunk boundary artifacts from the previous screenshot are gone!

Consider this drawing, where the gray blocks are one chunk (seen from above). We determine the four starting patches that overlap this chunk (the blue boxes). This needs to be random but deterministic, since neighbor chunks will need to use the same starting chunks. We query the pre-generated patches at the boundary of the chunk (shown in green) and use this as the boundary constraint for the generation of the chunk.

The green boundary area will stay the same during runtime generation, but this looks ok due to the variance in the pre-generated patches. The blue boundary is the same for each pre-generated patch, but will be replaced at runtime.

Note how this has the properties we want: Each chunk can be generated deterministically and independently from other chunks. If the generation for one chunk fails, we fall back to simply copying the blocks from the starting patches.

Using a heightmap

In this section, I'll explain how to generate a world in the shape of an arbitrary heightmap. This is done to achieve large-scale height differences, which are needed for a natural looking landscape.

Consider an integer heightmap where the difference between two adjacent points is always one. The next point is either one above or one below, but never at the same level or anywhere else.

Each 2x2 cell in that heightmap has one of these six shapes:

For each of these six possible 2x2 cell shapes, we pre-generate a set of starting patches:

These starting patches are no longer tiling in the classic sense. Instead, each side matches the opposite side with a vertical offset.

With our special integer heightmap where adjacent points always have a difference of 1, we will now generate one chunk for each point in the heightmap. Our query point has four adjacent 2x2 cells. For each 2x2 cell, we determine which of the six possible shapes it has and pick a pre-generated starting patch from the respective collection. Then, we generate a replacement map as explained in the previous section.

Here is an example of the heightmap in engine, each chunk is represented as one flat box:

This mesh is used to render the world far away from the camera. I added a "city like" texture and some billboard-rendered fake buildings. In the foreground, you can see the actual chunks generated by the algorithm:

Okay, now we know how to turn our integer heightmap into a cool looking infinite WFC world, but how do we get that integer heightmap in the first place? How do we get a function that generates an integer elevation function where the vertical difference between two adjacent points is always 1 or -1, but never 0 or anything else?

We start with a target function that doesn't have this property. In my case, I'm using 8 octaves of Perlin noise, but any heightmap can be used here.

Then, we use an elaborate clamping process to force the step constraint on our target function. It works in a hierarchical way, similarly to descending down a quadtree. We start with a relatively large square (the root of the quadtree) and evaluate our target function for the four corners. Then, we generate the heightmap value on the edge centers and the square center by querying the target function and then clamping the value to fulfil our slope constraint. The slope constraint requires that the vertical difference is less than or equal the horizontal difference. If our query point is inside any of the four quadrants, we repeat this process for the respective quadrant (descending the quadtree). If our query point is one of the points we just calculated, we're done.

The hierarchical nature of this approach means that it lends itself very well to caching.

Here is a 2D visualization of the process:

The blue line shows the target function. At every descend step down the quadtree, new limits are introduced to adhere to the slope constraint (shown as black lines). The orange dots are the values of our resulting heightmap.

Outlook and notes

Each chunk can be generated independently. That makes it easy to parallelize the computation required to generate the world. In my case, I'm using Unity's Burst compiler to do the runtime generation.

By varying the module probabilities for different areas of the map, I can generate different biomes. Here is an example of a biome boundary:

The biome on the left spawns copper roofs and bridges, the one on the right spawns tiled roofs and arches. On the boundary, there is one row of chunks where modules from both biomes can spawn, creating a natural looking transition.

I want to mention some progress on this project that is unrelated to the WFC algorithm. Since my last blog post in 2019, I've created lots of new blocks, textured them, added a water plane and added procedurally generated trees and climbing plants.

The trees and plants are generated at runtime using the Space Colonization algorithm and adapt to the geometry of the world.

The next challenge is to come up with interesting gameplay ideas for this project.

Adversarial Generation of Continuous Implicit Shape Representations

Sun, 02 Feb 2020 00:00:00 +0000

This article provides an overview of the paper "Adversarial Generation of Continuous Implicit Shape Representations", which I co-authored with Matthias Fey. While the paper focuses on the theoretical aspects, I'll provide a higher level explanation and and some visualizations here on the blog.

In the paper, we propose a GAN that generates 3D shapes. The GAN uses a DeepSDF network as a generator and either a 3D CNN or a Pointnet as the discriminator.

Update June 2020: Here is a recording of the presentation I gave about the paper at the virtual Eurographics 2020 conference.

DeepSDF

First, I'll introduce the idea behind DeepSDF. The standard way of representing 3D shapes in deep learning models uses voxel volumes. They are a generalization of images to 3D space and and use voxels instead of pixels. With this 3D CNN approach, concepts from deep learning for images can be applied to 3D shapes.

However, CNNs are well suited for learning texture properties and learning to represent a 3D shape as a combination of 3D texture patches is not ideal. Furthermore, due to the memory requirements of this approach, high voxel resolutions are not feasible.

A voxel volume contains a rasterized representation of the signed distance field of the shape. The signed distance function is a function that maps a point in 3D space to a scalar signed distance value. The idea behind the DeepSDF network is to train a neural network to predict the value of the signed distance directly for an arbitrary point in space. Thus, the network learns a continuous representation instead of a rasterized one. An SDF network for a single shape has three input neurons and one output neuron. The DeepSDF network doesn't use convolutions.

To learn a representation of multiple shapes, the DeepSDF network receives a latent code as an additional input.

The decision boundary of the network is the surface of the learned shape. For a given latent code, a mesh can be created using Marching Cubes by evaluating the network for a raster of points. The resolution of that raster can be selected arbitrarily after the network was trained.

The DeepSDF network is trained on a dataset of 3D points with corresponding SDF values. These points are in part uniformly sampled and in part normally distributed around the shape surface, resulting in a high density of training data near the surface.

Generating SDF data for the ShapeNet dataset is quite challenging because the dataset contains non-watertight meshes. I made my own implementation of the approach proposed in the DeepSDF paper, as well as a slightly different approach. I published this project as a python module.

I created my own implementation of the DeepSDF network and trained it on the ShapeNet dataset. The DeepSDF autodecoder works like an autoencoder, but without the encoder. The latent codes are assigned randomly and then optimized during training using SGD.

In the animation above, you see a t-SNE plot of the learned latent space of the DeepSDF autodecoder for the ShapeNet dataset. The colors show dataset categories which are not known to the network. It has learned on its own to assign similar latent codes to shapes of the same category. The shapes on the left are reconstructions of latent codes along a random spline path through the latent space. Click here for a high resolution version of the animation.

GAN

So far, I introduced the DeepSDF autodecoder that learns to reconstruct a given dataset of shapes. The contribution of our paper is to propose a GAN architecture that trains a DeepSDF network to generate new shapes.

A generative adversarial network is a pair of a generator network and a discriminator network. The generator proposes new samples (in our case shapes) and the discriminator predicts whether a given sample was generated by the generator ("fake") or taken from the dataset ("real"). The trick is that the generator is improved using the gradient of the discriminator output and the discriminator is trained on the dataset of real samples and the increasingly realistic fake samples provided by the generator. Thus, the generator and discriminator have adversarial goals. If the GAN training is successful, the GAN reaches an equilibrium where the generator has learned a mapping from the latent distribution to the underlying distribution of the dataset and discriminator assesses generated and real samples with the same score.

3D GANs based on 3D CNNs have been proposed. Our research question was whether a GAN can be trained where the generator uses the DeepSDF architecture.

Usually, the discriminator of the GAN is a mirrored version of the generator. In the case of the DeepSDF network, this is not feasible because a single sample of the DeepSDF network provides only the SDF for one point. From one point alone, a discriminator could not assess if the sample is realistic. Instead, the discriminator needs multiple points to judge the output value in context.

Voxel discriminator

One solution to the problem of missing context is to use a 3D CNN as the discriminator. In this case, the generator is evaluated for a batch of raster points and the resulting SDF values are rearranged into a voxel volume. The idea to combine a continuous implicit shape network with a 3D CNN was proposed by Chen and Zhang for their autoencoder.

The training data for the voxel discriminator are voxel volumes. We use datasets with resolution 8³, 16³, 32³ and 64³ and apply progressive growing. We use the Wasserstein distance with gradient penalty (WGAN-GP).

Here is a latent space interpolation of shapes generated with the voxel discriminator:

Another interesting observation is the generator's ability to generalize from the raster points to intermediate points that it was not trained on. Here is an example of a network that was only trained on 16^3 voxels (!), which is the first stage of the progressive growing.

On the left, the shapes were reconstructed with 16^3 raster points (the same resolution it was trained on). Scaling up the resolution linearly makes it smoother, but doesn't add any detail. When querying the network at 128^3, we see that the network is able to generalize to higher resolutions. For some parts of the geometry, the low resolution volume has no sample points with negative values, resulting in apparently missing geometry. In these cases, the network still generates negative SDFs for intermediate points, thus making the geometry appear when sampled at a higher resolution.

Pointnet discriminator

Since the voxel-based discriminator keeps some of the disadvantages of fully voxel-based GANs, we investigated an approach that doesn't use voxels at all. The Pointnet is a neural network architecture that can operate on point clouds. In our case, we sample a point cloud of uniform points and use the generator to predict SDFs for the points. The Pointnet then receives the positions of the points and the signed distance values as a "feature vector" with one element. This way, we avoid the fixed raster points and use always changing query points. A Pointnet typically infers information from the spatial structure of the points, which in our case is random. Regardless, we found that the Pointnet can be used as the discriminator in our case.

To train the GAN with the Pointnet discriminator, we use a ground truth dataset of uniformly sampled points with their corresponding SDFs. We "grow" the point clouds during training simply by increasing the number of points.

When the surface of the generated SDF is reconstructed using Marching Cubes, only values close to sign changes matter. We would like the network to spend more model capacity on values close to the surface, as they influence the result the most. We achieved that by refining the network with additional sample points close to the surface. The gradient of the SDF gives us the direction towards the shape's surface and for a neural network the gradient is easily computed. Using that, we can move randomly sampled points closer to the surface of the generated shape. The non-uniform point cloud can then be evaluated by the discriminator. Since the Pointnet takes the positions of the points into account, it could discern a uniformly sampled point cloud from a surface point cloud. Thus, we add surface points to the ground truth data as well.

Here are some examples of shapes generated with the Pointnet discriminator and uniformly sampled points (top) and shapes generated with the refinement method (bottom). (I'll replace this with an animated version soon.)

For both approaches, we found a surprising result when trying different levels for Marching Cubes. By default, Marching Cubes reconstructs the isosurface at the level 0, i.e. the surface where the predicted SDF is zero. However, we can choose another level, effectively eroding or dilating the shape. We would expect that the ideal level would be 0, since that is the isosurface of the training data. However, when experimenting with different values, we observe that some features appear to be missing at level 0.

We chose an isosurface level of 0.04 to reduce missing geometry at the cost of slightly rounded corners. Since we clip the ground truth SDF at -0.1 and 0.1, the isosurfaces of generated SDF outside of that range are not usable.

The source code for this project is available on Github.

What I learned from building autonomous model race cars for a year

Fri, 03 Jan 2020 00:00:00 +0000

I was part of a university project group that develops autonomous model race cars. We are a group of twelve students working on the project in part time for year.

We were provided with a car that meets the requirements for the F1/10th competition. Even though competing in F1/10th was not our goal, we kept the rules for the competition in mind. We focussed mostly on trying different driving algorithms, which I'll explain below. The software we developed is available on Github.

This post is about what we did and what insights I gained from this work.

Hardware

Our car is a regular 1/10 scale RC car where the the RC receiver is replaced with a computer. The car is based on a Traxxas Ford Fiesta ST Rally. The computer is an Nvidia Jetson TX2. The car is equipped with a LIDAR scanner with a resolution of 1 ✕ 1080, an IMU and a stereo camera. We ended up not using the stereo camera.

ROS

Our software stack is based on ROS and Gazebo. We used ROS and Gazebo because it is the standard for academic robotics projects. Most online materials assume that your project uses on ROS.

ROS is a robotics framework that facilitates communication between robot compontents such as sensors and actors. All the hardware components we use have ROS nodes that publish or subscribe to data in a standardized way. ROS provides a rich ecosystem of software. For all kinds of robot parts and control problems there are ROS nodes available. You can get pretty far by just tying together existing ROS packages. ROS nodes can be written in C++ and Python 2.7 and since each node is its own process, both languages can be used at the same time.

A ROS architecture is built like a distributed system with nodes that subscribe and publish to topics. This is the case even if everything runs on a single computer, like with our project. This creates some overhead. Using ROS means that you'll write lots of configuration files, package definition files, launch files, robot description files that all reference each other. With Gazebo, you'll also need mesh description XML files and world definition files. However, I think that using ROS is worth it for ecosystem of existing ROS modules.

Gazebo

We use Gazebo as a simulation environment. Our robot car can drive in the real world and in the Gazebo simulation using the same software. Having this simulation turned out to be extremely useful. Since we mostly used the Lidar, the simulated track is designed with walls enclosing the track. The visual appearance and realism was less important for our use case.

What follows next is my rant about Gazebo. Gazebo is bad software in every way. It is poorly documented, buggy and misses important features. For example, about 20% of the time, Gazebo doesn't launch at all and crashes instead. In the video above, you can see that the car receives shadows, but the track doesn't. See how the car isn't shiny? That's because Gazebo's lighting model doesn't support metalicity. Getting it to render the scene this decent was super difficult, it would be trivial to get way better looks in an engine like Unity. These are just a few out of many problems we had with Gazebo. Seemingly trivial things are unneccesarily difficult.

My takeaway is: Avoid using Gazebo if at all possible. Use a game engine instead. Getting a working game engine to simulate a robotics scenario is way easier than getting Gazebo to do what it's supposed to do. For example, there is a project to let ROS communicate with the Unity engine. This is what you should use instead, it will save you a lot of headaches. There are some features in Gazebo specific to robot simulation that a game engine doesn't provide, such as a headless mode.

Now for the interesting part, the autonomous driving algorithms. All algorithms use only LIDAR and IMU data. We didn't use the camera.

SLAM

SLAM stands for simultaneous localization and mapping. It is the concept that the robot determines its own position in an unknown environment based on a map that it creates while exploring the enviroment. For autonomous racing, this means that the car generates a map of the racetrack and can then calculate and follow an ideal racing path using that map and its position.

Since we use ROS, we can use lots of existing SLAM implementations that are built specifically for ROS. In addition, ROS offers the navigation stack, which has implementations for path planning and execution. But working with SLAM in practice turned out to be difficult. The existing SLAM algorithms are mostly designed for slow robots and stop working at higher speeds. This makes them unsuitable for racing. But seeing the map generate looks pretty cool!

Wallfollowing

Wallfollowing is what we call our greedy driving algorithm. It has no concept of the track or the ideal path. It only considers the part of the track that the LIDAR can see from the current position of the car.

Our approach is to separate the laser sample points (shown in red) into a left and a right wall and fit two circles into them. The algorithm calculates a predicted position of the car (in the center of the video) and a desired position (in the center of the track). The difference between them (shown in orange) is used as the error for a PID controller, which controlls the steering. To control throttle, we calculate multiple maximum speeds based on the PID error, the curviness of the track and the distance towards the next obstacle (shown in yellow). Out of these maximum speeds, we apply the smallest one. This allows us to slow down in curves, cut corners, etc.

Teams competing in the F1/10 competition typically use an approach like this. The wallfollowing approach provided the best lap times in our tests.

The videos for Gazebo and for SLAM above show the car being controlled by the Wallfollowing algorithm. Here is a video of the physical car driving with it:

Reinforcement Learning

One of the methods of autonous driving we tried during this project group was reinforcement learning. It is an area of machine learning where an agent, in our case a car, learns to select actions given a state to optimize a reward function. In particular, we used deep reinforcement learning, where the function that selects an action given a state is a neural network, instead of a lookup table.

In our case, the state vector was a downscaled version of the Lidar scan. That means, each element of the vector contains the measured distance for a fixed direction. We also experimented with other state information, such as velocity and throttle, but this didn't bring any improvement. In reinforcement learning, the actions are discrete, meaning that the agent selects an action out of a finite set, instead of providing numbers for throttle and steering. In the simplest example, we used a fixed velocity and two actions for steering left and right. Straight driving would be achieved by oscilating between left and right. We also tried more granular action sets, but this increases the difficulty of the learning task. The neural network has one input neuron for each element of the state vector and one output neuron for each action. It predicts a Q value for each action and the car will perform the action with the highest Q value. During training however, sometimes random actions are taken instead, according an epsilon-greedy policy (exploration vs. exploitation). For the reward function, it would be possible to use a reward of 1 for each step. There is an implicit reward for not crashing as this leads to longer episodes and the reward is accumulated over the episode. But it will learn to stay in the center faster if we give it higher rewards for driving close to the center of the track and lower rewards for driving close to the walls. In addition, we reward high speed.

We tested Q-Learning and Policy Gradient. Both took hours to learn to drive in the center, but eventually did so decently. Overall, policy gradient worked better than Q-learning. But both were unreliable in the simulation and didn't translate well from simulation to reality. I think that we could achieve significantly better results if we had more time.

This is a video of the car driving in our simulation with policy gradient:

Evolutionary Algorithm

For the evolutionary algorithm, we took the neural networks from our reinforcement learning approach and learned the network parameters with an evolutionary algorithm. Instead of using a set of actions, we let two output neurons control throttle and steering directly. We started with random weights and randomly changed them while measuring the performance of the car in the simulation.

Surprisingly, this worked about as well as reinforcement learning, but was easier to implement and took less time to train. However, like reinforcement learning, it did not perform as well as our wallfollowing algorithm.

Outlook

Since our goal was not primarily to participate in the official competition, we investigated sevaral methods of autonomous driving. The least fancy one, our greedy wallfollowing algorithm, turned out to be the fastest. All of the driving methods could probably be improved with more time. There is a follow-up project group that continues to work with our code base. It looks like they are even working on moving away from Gazebo.

Visualizing 150000 butterflies from the Natural History Museum

Thu, 26 Dec 2019 00:00:00 +0000

Click here for the interactive visualization.

The Natural History Museum in London has a data portal in which they provide digital records for many of their specimens. Some of these records have images. I recently learned how to use machine learning tools such as convolutional neural networks and I wanted to use the NHM data to see what can be done with these tools. The dataset of butterflies seemed particularly interesting to me because the images are visually interesting, yet they are all similar in that they all contain a butterfly in a canonical pose.

Since I'm only interested in learning the visuals of the butterflies and not the rest of the photographs, such as the labels, I needed to find a way to separate the butterflies from the background. Here, you can see one collection of ~2000 butterflies, which are photographed against a clear background:

It is quite easy to remove the background automatically based on contrast. However, besides this collection, there are a lot more images of butterflies available. They can be found by searching for "lepidoptera" in the data portal. This way, I downloaded 30 GB of butterflies. Most of the 150000 images have noisy backgrounds with varying brightness, such as this one:

To separate the butterflies from the backgrounds, I trained a U-Net that predicts for each pixel of the image whether it belongs to the butterfly or background. It is a convolutional neural network that receives an image as the input and produces a new image as the output, in my case a background-foreground mask.

I started by manually creating masks for a few images:

I trained the U-Net and tested it on random images from the dataset. From those I selected the images where the classification went poorly and manually added masks for them to the training set. This selection process leads to a ground truth dataset that contains mostly "challenging" items. I ended up with ~300 masks, however it works reasonably well even with fewer images. Since the network needs to make a prediction for every single pixel, it is forced to generalize even when trained on a single image.

Using the masks generated by the U-Net, I can remove the background and crop all the images in the dataset:

Next, I created a 128✕128 image with a white background for each butterfly and trained an autoencoder on them. The autoencoder receives an images as the input and produces a reconstructed image as the output. Its loss function is the difference between the input and output so that during training, it learns to reduce that difference and creates an accurate reconstruction of the input. The autoencoder has a bottleneck in the middle which consists of 128 neurons. This way, it learns to compress each image into a latent vector of 128 numbers.

Here are some examples of images reconstructed with the autoencoder:

During 6 hours of training, the network didn't converge, meaning that a better quality could be achieved by training longer. However, for this project I'm mostly interested in the latent space learned by the autoencoder, not the reconstructed images.

I used the encoder of the autoencoder to calculate the latent vectors for all images in the dataset. To visualize the latent space, I used t-SNE to transform the latent vectors from 128-dimensional space to a 2D plane.

In this t-SNE plot, each dot corresponds to an image in the dataset. The t-SNE transformation places the points so that those points appear close to each other that have similar latent vectors, meaning that they appear similar to the autoencoder.

Next, I wanted to replace the dots in the plot with images of the corresponding butterflies. To prevent overlapping of very close points, I iteratively moved them apart. This was done by defining a "safe" radius and finding all points that have a neighbor within this radius. Each point is then moved away from its closest neighbor by a small distance:

Here is a part of the resulting image, where I also added some drop shadows:

We can already see that similar butterflies are clustered together. However, this t-SNE plot is 131000✕131000 pixels big. That is 17 Gigapixels! To display it, I'm using Leaflet, a javascript library that can render maps. The image is split into tiles with a resolution of 256x256 pixels which are loaded dynamically by Leaflet. I also created tiles for lower zoom levels, so that people can zoom out like on a map.

When the map is zoomed all the way out, the images appear as tiny dots. For the intermediate zoom levels, I rendered a few "representative" images. These are selected by applying a k-means clustering to the t-SNE points. For each cluster, a representative image is rendered at the cluster center. The representative image is selected by calculating the average of the cluster in latent space (not t-SNE space) and finding the butterfly with the latent vector that is closest to the average. I found that when using t-SNE space to determine the representative, an outlier is often selected, which doesn't happen in latent space.

The tileset contains 138000 images, which is ~900 MB. This is what it looks like to zoom around in the interactive t-SNE plot:

Click here for the interactive version of the t-SNE plot.

I also added the ability to click on the butterflies to show a popup with the original image, the scientific name and some other data.

These information are provided in the NHM data portal as a CSV file. I wrote a script that creates a JSON file with the t-SNE positions of the points and the corresponding data. But this metadata file alone is 50MB in size. It is not feasible to load it when someone visits the website. So I split the metadata into smaller chunks using a quadtree, much like the images are split into tiles. Now the metadata is loaded only for the regions that the user looks at.

Using the data from these CSV files, I can color the dots in the t-SNE plot according to the genus of the butterflies:

We can see that the autoencoder has learned to separate the butterflies by genus, solely based on the visuals, without being provided any information about the genus! This is an example of unsupervised training of a classifier.

Similarly, it mostly placed the families in adjacent clusters:

There is an area in the plot where lots of colors mix. These are specimens with missing wings. Since the missing wing is the most striking visual feature, it primarily determines the latent vector that the autoencoder assigns.

Using the same method for the sex, we can see that most of the records contain no sex information. But for the birdwings, for which this information is available, the autoencoder has learned to separate male from female, again, without being tasked to do so.

We can observe a similar result for the delias.

The source code for this project is available on Github. The images of the butterflies are provided by the Trustees of the Natural History Museum under a CC BY 4.0 license.

Lego Part Designer

Wed, 01 May 2019 00:00:00 +0000

I made a web app that lets you design your own Lego Technic parts and save them as printable STL files. You can check it out here.

I got the idea for this project when I was building with Lego parts and wondered how many of the common parts can be described with a simple rule set. The system I came up with assembles the parts out of five basic blocks: Pin Hole, Axle Hole, Pin, Axle and Solid.

Internally, each block consists of four small blocks. By default, double sized blocks are created that consist of eight small blocks. All blocks are placed using coordinates of the small blocks so that they can be placed at "half" positions.

To accomodate 0.2 mm margins at the outside faces of the part, the small blocks are split further into three tiny blocks per dimension. The two outer tiny blocks have the size of the margin and the center tiny block takes the remaining space. Note that a tiny block can be in the center on one axis and in the margin on another. The code has some logic to determine which tiny blocks are part of the model and which aren't. Finally, before rendering the tiny blocks, they are merged to lower the number of polygons.

The app is written in TypeScript and uses WebGL to render the model. The difficult part is generating the mesh for the model. This is done directly in the TypeScript code, without using a CSG library. There are no big surprises in the implementation, just lots and lots of corner cases to handle, many of them literal corner cases. The source code for this project is available on Github.

One problem with TypeScript is its lack of support for operator overloading. This project uses lots of vector operations and without operator overloading, I can't write v1 = v2 + v3 * f;, instead I have to write v1 = v2.plus(v3.times(f));. This can make the code unnecessarily cluttered.

The app features a catalog that shows a selection of existing Lego parts that can be made with this system. Of course the point of this tool is to make parts that don't already exist, but this can give you some inspiration.

The preview images for the catalog are not stored on the server. Instead, they are rendered using the same code that renders the model in the editor.

Orrery gear box

To demonstrate the practical use of this app, I made a gear box for a Lego orrery. The orrery is based on a design from JK Brickworks. I took the sun, earth and moon design from that model and designed everything else myself.

Here is the gear box in the part designer:

After printing and adding gears, it looks like this:

And finally, here is the completed orrery:

The gears run reasonably smoothly in the printed part. Choosing the size of the axle holes is a tradeoff between having too much resistance and being too loose. Of course the real, injection molded parts have tighter tolerances than one can achieve with a consumer 3D printer. This example shows that 3D printed parts can be combined with original Lego parts. One advantage of printing parts like this is that it saves you the time to figure out how a structure can be made with existing Lego parts.

Faster Than Life – Global Game Jam 2019 Project

Sat, 02 Feb 2019 00:00:00 +0000

Like in the previous year, I took part in the Global Game Jam. I joined a team of six programmers, unfortunately there was a shortage of artists this year. During the 48 hour jam, we made a space game that is inspired by Faster Than Light.

You travel through a sector of solar systems, with the goal to make your way home. In each system, you need to collect jump fuel to jump to the next system. You can die by crashing into a planet or by flying too far into deep space. The amount of fuel in the spacecraft is also limited, but it regenerates over time.

My contribution was making the planets move on somewhat realistic, elliptical orbits. I wanted to achieve the same for the spaceship, but didn't manage to do it in time. So now it has semi-realistic physics, but it's influenced by all bodies in the system. I hope that I'll do more with realistic orbits in Unity in the future.

Our team put quite some effort into making procedurally generated low-poly planets. These ended up being quite small in the levels, so we added "personal" planets to our credit screen:

You can download the game on the Global Game Jam website or watch me play through it below.

Infinite procedurally generated city with the Wave Function Collapse algorithm

Sun, 06 Jan 2019 00:00:00 +0000

This is a game where you walk through an infinite city that is procedurally generated as you walk. It is generated from a set of blocks with the Wave Function Collapse algorithm.

You can download a playable build of the game on itch.io and you can get the source code on github. Here is a video of me walking around a generated city:

The algorithm

I will use the word "slot" for a place in the 3D voxel grid that can contain a block (or be empty) and I will use the word "module" for a block that can inhabit such a slot.

The algorithm chooses which modules to select for each slot in the world. The array of slots is considerd a wave function in an unobserved state. That means that each slot has a set of possible modules that could be put there. In the language of quantum mechanics, one could say "The slot is in superposition of all modules". The world starts in a completely unobserved state where every module is possible in any slot. One by one, each slot is collapsed. That means, one module from the set of possible modules is selected at random. This is followed by a step of constraint propagation. For each module, only a subset of modules are allowed to be placed adjacent to it. Whenever a slot is collapsed, the sets of modules that are still possible to be placed in nearby slots need to be updated. The constraint propagation step is the most computationally expensive part of the algorithm.

An important aspect of the algorithm is deciding which slot to collapse. The algorithm always collapses the slot with the lowest entropy). That is the slot which has the least amount of choice (or chaos). If all modules have the same probability, the slot with the smallest number of possible modules has the lowest entropy. In general, modules have different probabilities to be selected. A slot with two possible modules of the same probability has more choice (greater entropy) than one with two modules where one is very likely and one is very unlikely.

(Gif by Maxim Gumin on Github)

You can find more information and some beautiful examples of the Wave Function Collapse algorithm here. The algorithm was proposed to generate 2D textures from a single example. In that case, the module probabilities and adjacency rules are determined based on how they occur in the example. In my case, they are supplied manually. Wave Function Collapse is based on Paul Merrel's Model Synthesis.

Here is a video of the algorithm in action:

About blocks, prototypes and modules

The world is generated from a set of ~100 blocks, which I made with Blender. I started out with a small number of blocks and made more whenever I had some spare time.

The algorithm needs to know which modules can be placed next to each other. Each module has 6 lists of possible neighbors, one for each direction. But I wanted to avoid having to create this list manually. I also wanted a way to automatically generate rotated variants of my blocks.

Both can be achieved by using what I call module prototypes. This is a MonoBehaviour that can be conveniently edited in the Unity editor. The modules together with lists of allowed neighbors and the rotated variants are created automatically from these.

A difficult problem was figuring out how to model adjacency information so that this automatic process works. Here is what I came up with:

Each block has 6 connectors, one for each face. The connector has a number. In addition, horizontal connectors are either flipped, not flipped or symmetric. Vertical connectors either have a rotation index between 0 and 3 (b, c, d in the screenshot) or they are flagged rotationally invariant.

Based on this, I can automatically check which modules are allowed next to each other. Adjaced modules must have the same connector number. And their symmetry must match (same rotation index vertically, a flipped and not flipped pair horizontally) or they must be symmetric / invariant.

There are exclusion rules that allow me to prohibit neighbors that would otherwise be allowed. Some blocks with matching connectors just don't look nice next to each other. Here is an example of a map generated without the exclusion rules:

Reaching infinity

The original Wave Function Collapse algorithm generates finite maps. I wanted to have a world that expands further and further as you walk through it.

My first approach was to generate chunks of finite size and use the connectors of adjacent chunks as constraints. If a chunk is generated and an adjacent chunk was already generated, only modules are allowed that fit with the existing modules. The problem with this approach is, whenever a slot is collapsed, the constraint propagation will limit the posibilities even a few slots away. In this image you can see all the places affected from collapsing just one slot:

When just generating a single chunk at once, constraints were not propagated to adjacent chunks. This led to modules being selected within the chunk that would not be allowed when considering the other chunks. When the algorithm would then try to generate the next chunk, it could not find any solution.

Instead of using chunks, I store the map in a dictionary that maps a slot position to a slot. It is only populated when needed. Some parts of the algorithm needed to be adjusted to this. When selecting a slot to collapse, not all infinite slots can be considered. Instead, only a small area of the map is generated at once, when the player reaches it. Constraints are still propagated outside of this area.

In some cases this approach doesn't work. Consider a module set with the straight tunnel piece from the screenshot above, but no tunnel entrance. If the algorithm selects such a tunnel module, this predetermines an infinite tunnel. The constraint propagation step would try to allocate an infinite amount of slots. I designed the module set to avoid this problem.

Boundary constraints

There are two important boundary constraints: Faces at the top of the map must have "air" connectors. Faces at the bottom of the map must have "solid" connectors. If these constraints are not met, there are holes in the ground and buildings with missing roofs.

In a finite map, this would be easy to do: For all slots in the top and bottom layer, remove all modules with unwanted connectors. Then use constraint propagation to remove other modules that are no longer valid.

In the infinite map, this doesn't work because there are infinitely many slots in the top and bottom layer. Naively, I would only remove these modules in the top and bottom layer once the slots are created. However, removing a module in one top layer slot implies constraints for its neighbor slots. This leads to a cascading effect which would again allocate slots infinitely.

I solved this by creating a 1×n×1 map, where n is the height. This map uses world wrapping to propagate constraints. This works like Pacman, you leave the level on the right and enter on the left. Now in this map I can apply all the boundary constraints. Whenever a new slot in the infinite map is created, it is initialized with the module set of the corresponding position in that map.

Error states and backtracking

Sometimes the WFC algorithm will reach a state where a slot has zero possible modules. In applications with a finite world, you can just discard the result and start over. In the infinite world, this doesn't work, since a part of the world has already been shown to the player. I started with a solution where a white block would be spawned in places of errors.

My current solution is backtracking. The order in which the slots where collapsed and some information about constraint propagation is stored as a history. If the WFC algorithm fails, some of the history is undone. This works in most cases, but sometimes errors are recognized very late which leads to many steps being backtracked. In rare cases, the slot in which the player is, is regenerated.

In my opinion, this limitation makes the WFC approach for infinite worlds unsuitable for commercial games.

Outlook

I started working on this when I saw a talk by Oskar Stålberg who uses the WFC algorithm to generate levels in Bad North. Most of the basics were implemented during the procjam week.

I have some ideas about future improvements, but I'm not sure if I'll ever add gameplay. And if I do, it will probably not be the battle royale game that you are envisioning. But if you want to see your favorite game mechanic added to this, just do it yourself! The source code is available after all and it's MIT licensed.

ESA ExoMars Rover 3D model

Sun, 22 Jul 2018 00:00:00 +0000

This is one of my first 3D modeling projects in Blender and my biggest 3D modeling project so far.

It's a model of ESA's ExoMars rover. You can have a closer look at the model on Sketchfab:

There is also a download option on Sketchfab so you can get the original .blend file and use it for anything you like!

3D printed model of my neighborhood

Tue, 17 Apr 2018 00:00:00 +0000

I 3D printed a model of the street where I live. This post will explain how I prepared the data for it.

Update: I have now automated the entire process and published my code. You can find it here.

I worked with aerial Lidar data that is provided by the state I live in to download for free. After completing the rather big download, I figured out what my coordinates are in the projection used with the data and which XYZ file they fall into. I wrote a simple program to extract all the points within a 200m square around my house.

This left me with a smaller XYZ file, now containing 430k points, which are pictured above. The task now was now to turn all these points into a solid body in order to be printable.

In order to turn the pointcloud into a mesh, I used the software MeshLab. As a first step, I used the program to estimate normals for the pointcloud. The normals are estimated by fitting a plane into some nearby points and using the normal of that plane. However, knowing the plane, there are still two possible normals of which one has to be selected. For some separated parts of the pointcloud, the normals can be flipped and need to be manually fixed.

Here is a mesh that MeshLab generates without normals fixed and wihtout holes patched:

Since the Lidar data are taken from an airplane, parts of the surface that are occluded from the perspective of the plane result in holes in the pointcloud. After trying to generate a mesh with no success, I had to fix the holes first.

I made a program to detect and fill the holes. First it tries to identify points on the edge of holes. This is done by measuring the distance of a point to the average of some nearby points. This distance is small when the nearby points are found in all directions. Points on an edge have less neighbours towards the hole, resulting in an average shifted away from the edge. Points found like this are shown in red in the above screenshot.

Next, to create patches, the program looks for edge points with other edge points straight above them. Depending on the distance, new points are then created on the connecting line. These are shown in green.

Now I can apply what MeshLab calls “Screened Poisson Surface Reconstruction”. With the holes patched, normals corrected and parameters tweaked, I have a mesh of the surface.

This mesh can be opened in Blender. I selected the outer vertices, moved them down and connected them, resulting in this:

This is finally a solid body, although an ugly one. I created a 200m cube and applied a Boolean / Intersect operation between the two, resulting in the final model for printing. Here it is being prepared in Cura:

I printed it 12cm long with the smallest layer size available, 0.06mm, which took 21 hours.

Agent V – Global Game Jam 2018 Project

Sun, 04 Feb 2018 00:00:00 +0000

This year I participated in my first game jam, the Global Game Jam 2018. With a team of artists, programmers and a sound designer, we made a video game within 48 hours. You play the game as a virus that infiltrates a company’s headquarters. The virus can not move on its own, it can only take control of hackable devices and use them to move around. There are six levels, each with a puzzle to move to the next one.

You can download the game on the Global Game Jam website or watch the playthrough below.

Generating 3D roof meshes from aerial LIDAR data

Tue, 02 Jan 2018 00:00:00 +0000

This is my graduation project I did in computer science. The goal was to come up with a method to generate 3D meshes of building roofs from point cloud data. The point cloud data was taken with aerial LIDAR scanners and is available online. In addition, I used building layout polygons, which are also available as open data. I tested several strategies to generate a mesh and I’ll explain the best one in this post.

Separating the point cloud by buildings

The dataset for the city of Dortmund contains about 4×10^9 points. For this city, the building layout dataset has about 200,000 entries. My first step was to write a program that finds the subset of points that belong to each building.

As the building layouts are polygons, the core of this operation is the point in polygon test. The important takeaway for this part is to use spatial hashing. The polygons are stored in a hash set so that for each point, only a few polygons need to be considered. It took four hours to process 100 GB of data on my laptop.

Finding planes

The next step is to look for planes that describe many of the points in a point cloud. These planes are likely to portray roof faces, which makes them a good foundation for generating a roof mesh.

It turned out that a RANSAC approach works well for this problem. The idea of RANSAC is to select a subset of points at random and generate planes from these sample points. To get a sample plane from a sample point, a fit plane for nearby points is calculated. For each sample plane, a score is calculated that tells how well the plane fits the entire point cloud. The best scoring planes are returned.

The picture below shows an example point cloud with planes found by the RANSAC algorithm.

Algorithm to generate convex roof meshes

Now that we have a set of points that belong to a single building, the building’s layout polygon and a set of planes, it’s time for a mesh generation algorithm. The idea of this algorithm is to start with an infinetly extruded layout polygon and cut away each of the roof plane. Each cut creates a roof face.

Here are some examples of roof types that can be modeled this way:

The algorithm to generate these meshes works a bit different from the explanation above. For each roof face, the layout polygon is triangulated and projected onto the plane. In a second step, this mesh is cut along all roof planes, leaving only the geometry that lies below all other planes. All roof faces are generated like this and are then combined.

This graphic shows a run of this algorithm with a simple gabled roof. Two roof planes result in two roof faces. For each plane, there is only one other plane that is cut away from the mesh.

The choice of roof planes is quite important, as the RANSAC algorithm usually returns more than the desired set of planes. To find the best ones, each possible subset of planes is tested. Each time, a roof mesh is generated and scored. The best one is returned.

Here are some examples of meshes generated with the above algorithm:

Attachments

As the name suggests, the algorithm to generate convex roofs is limited to convex roofs. In the real world, many roofs aren’t convex, like these ones:

To generete roof meshes for these roofs, I extended the algorithm to look for attachments. It starts with a convex roof. For each face, the subset of points above the roof plane is considered. The same algorithm is used to generate meshes for attachments. These attachments are added to the resulting mesh.

A postprocessing step is needed to avoid some unwanted geometry that arises from combining the base mesh with the attachment meshes. Sometimes, this step introduces unwanted artifacts in the result.

Here are some exmaples of convex roofs with attachments:

Implementation

I used the Unity game engine to test the algorithms. Unity makes it easy to display meshes and point clouds. For common geometry problems like triangulating polygons, there is code available online.

You can find the source code for the Unity implementation on github.

Data sources

The point data used in the screenshots in this article was calculated using point data and shape data from https://www.opengeodata.nrw.de/ Source for point data: https://www.opengeodata.nrw.de/produkte/geobasis/dom/dom1l/dom1l_05913000_Dortmund_EPSG25832_XYZ.zip Source for shape data: https://www.opengeodata.nrw.de/produkte/geobasis/lika/alkis_sek/hu_nw/hu_nw_EPSG4647_Shape.zip The data was provided by the "Land NRW" under the Data licence Germany – attribution – Version 2.0 (dl-de/by-2-0) Full license text is available at http://www.govdata.de/dl-de/by-2-0 The screenshots in this article contain map data by OpenStreetMap (ODbL) and Positron tiles by CartoDB (CC BY 3.0).

LED Matrix Software

Sat, 13 May 2017 00:00:00 +0000

I wrote two programs that run on my LED matrix. They have different approaches and different aims. This post describes one of them.

The app offers a web interface where users can write simple programs that are then compiled and started instantly from the website. The concept is similar to Shadertoy, where you write C-like shader code in a browser window and instantly see it run. The code of for this project is on my github and it’s in a state where you should be able to run it fairly easily, if your hardware is similar. The github page has installation instructions and some documentation on the available functions for user apps.

The backend for the webapp is written in Python, using flask. The user apps are written in C++. I first tried to use Python for this as well, but it wasn’t fast enough. Just iterating through a 256 color list already impacts the framerate. The C++ code is compiled into a shared object library file. The library is loaded into the python app using the python module ctypes. The client is written in Javascript, using codemirror to edit code and bootstrap for the UI.

There is a gif recording feature that creates live recordings of the apps you write:

LED Matrix Materials Guide

Mon, 08 May 2017 00:00:00 +0000

I built an LED matrix out of 256 WS2812 LEDs. This post will describe which materials I used and which I tried with no success so you don’t have to.

Case

For the case, I used a custom made photo frame. The main purpose of the case is to look good, which is harder to achieve if you make it yourself. I’m using a matte black aluminium frame. Make sure it is deep enough to house all parts. My frame came with a glass plate, so I didn’t need to buy a seperate one.

I used a sheet of scrap wood as a base plate to stick the LED strips on and to mount the other components.

Dividing grid

You need a dividing grid in order to prevent neighbour pixels from blending colors. Without a dividing grid, there will be no sharp edges between two pixels, but a gradient, making the resulting image look blurry.

My first try was making a grid of interlocked cardboard strips. The resulting walls bent, creating squiggly pixels.

Next, I ordered wood strips, cut notches in them and made another interlocking grid. This resulted in the same bending as the cardboard approach. Even though the wood strips bend less, small inaccuracies with the positioning of the notchces lead to tension and bend the pixels.

To get rid of the inaccuracies of manual measuring, I tried a laser cutter. My first approach was to cut the entire grid out of multiple layers of wood. The walls were too thin and the laser destroyed them. Since I only had access to the laser cutter for a limited time, I didn’t explore it exhaustively. You might get better results with the interlocking approach or with a different material.

Finally, I got a 3D printer and printed a grid. This is the one I ended up keeping for my matrix. I had to split it into four parts to fit in the printer’s build volume. If your grid model has bridges for the LED strips, you might try to print the grid with the viewer facing side pointing up in the printer. However, the unwanted flatfood effect will be visible in the LED matrix if you print your grid like this. Instead, print it so that the viewer facing layer is printed last.

Diffuser

In order to light up the entire pixel, not just a small dot in your field of view, you need something to diffuse the light.

I found a sheet of diffused acrylic glass. This is not suitable for using in an LED matrix. The thicker the diffuser is, the more it blurs the image, in addition to diffusing it.

This is a thinner sheet which creates a slightly sharper image. However, for a good image, the diffuser needs to be as thin as possible.

Looking for something thinner, I ordered diffusing 0.5mm thick plastic sheet from a crafting supplier (1). This still creates a bit of blur.

Next, I tried paper (2). I used vellum / transparent paper, which is sold for crafting. While still a bit too thick, and thus making the image a tiny bit blurry, the main problem was that the paper got rippled after a while. This adds an unwanted texture to the LED screen.

Finally, the solution I sticked with is a plastic foil. These are made to be put on windows for privacy. I found a website that sells these foils and offers free samples, so I got a selection of samples. Sample 3 and 4 don’t diffuse the light enough, you can make out the shape of the LED. Finally, I got sample 5 and 6 in full size. I found the non adhesive foil easier to handle. It stays in place well without being glued to the glass plate.

Electronics

Depending on the software you intend to run on your LED matrix, I recommend using a Raspberry Pi or, for simpler projects, an Arduino. If your board operates at 3.3V, you might want to use a level shifter that converts the 3.3V signal to the 5V signal the WS2812 strip needs. However, if you feel adventurous, you can skip the level shifter and hope that the WS2812 picks up the 3.3V as a logical 1, which it will probably do.

In addition, my setup has four buttons and a microphone, these are optional depending on your intended use of the LED matrix.

Adafruit has a good tutorial for setting up the LED strip and the software for it on a Raspberry Pi. You should make sure to get the strandtest.py example running before sticking the LED strips.

LEDs

I’m using WS2812 LED strips. As long as the model is correct, just go for the cheapest offer. Check the estimated shipping time when ordering from China.

LED layout

The LEDs are supplied as a single strip and will appear as such to the computer. The LED strip will be cut after every line and then placed with alternating orientation for each line of the grid. Since cutting and resoldering every pixel is tedious, you should consider the spacing of the LEDs when selecting the LED strip and chosing the spacing for your grid. I used a strip with 60 LEDs per meter. The 16 pixels plus a margin of one pixel on each end results in a 30cm x 30cm base plate. This way, I don’t need to cut after every LED, but only once per row.

Note that the LEDs are not centered between the cutting lines. You need to make sure that the LEDs themselves line up, not the cutting lines. I realized this too late.

Power supply

Each subpixel LED draws 20mA at full brightness. That’s 60mA per pixel or 15.3A for 256 pixels.

Going by that math, you’ll need a power supply that can supply at least 15A. However, this only really applies if you want to have your entire LED matrix on bright white for long amounts of time. In practice, a lower rated power supply will do the job. Adafruit proposes that as a rule of thumb, you can get away with 1/3 of the maximum theoretical power draw. This would be 5A in the case of 256 pixels. I’m using a 10A power supply for my project and I didn’t have any issues.

However, operating a power supply beyond specification can be a fire hazard.

Also note that running 256 LEDs at their brightest level is really bright. Unless you need a floodlight, you might want to limit the brightness anyway.

Additional hardware

Cosmetic grid

This is a thin grid made out of black adhesive foil with a cutting plotter. It is applied to the glass sheet and its purpose is to give the pixels a visual seperation, creating a sharper looking image. This is optional and I’m not entirely sure if I like the matrix better with or without it. It looks bad at places where the dividing walls behind the diffuser don’t line up with the cosmetic grid. When turned off, the grid does make it look more interesting.

Electronics case

I 3D printed a case for the electronics. This is also optional, since the matrix works perfectly well without it, but it looks cool. It also contains an USB port for the Raspberry Pi Zero W I use, so I can connect a gamepad.

No Man’s Starfield

Sun, 18 Sep 2016 00:00:00 +0000

This is a shader I made that renders a flight through a starfield. It’s meant to look like the galactic map that you can see in No Man’s Sky. Here is a link to the project on shadertoy:

No Man’s Starfield

The cool thing about this is that it runs inside a shader. A shader is a small program that runs on the graphics card and is usually used to create simple effects like shadows or reflections. However, it can also be used to make really complex things. In a shader, all pixels are computed separately and in paralell. That means the program gets the pixel coordinate and computes a color for it. This program is then called for each pixel in the frame.

To render 3D worlds, realtime graphics apps usually use depth buffers. However, inside a shader, one is forced to use raytracing. This means that for each pixel, a virtual light ray is followed from the camera until it hits something.

In this case, I need to check if the ray hits a star. Computing the distance between every ray and every star to check if it’s less than the star radius would take too long. Instead, the ray travels through a 3D grid. Every grid cell either has a star or not. This is determined using simplex noise. With this technique, the ray needs only to be compared to the stars within the grid cells that the ray travels through, and only those grid cells that contain a star.

Using raytracing also makes it easy to bend rays. My shader uses this to bend light rays close to black holes. Black holes occasionally appear instead of stars.

Computer Vision and Robotics Demo with Raspberry Pi

Sat, 02 Jul 2016 00:00:00 +0000

This spring, I spent some time at SAP’s commercial hackerspace. I wanted to explore how computer vision can be used with embedded devices and robotics. I built a demo that can detect QR codes and similar symbols and point a laser at them. Possible applications of this are putting QR codes on objects to help the robot locate them and grab or manipulate objects. Another possible use case is local navigation. A robot could infer its own location and orientation in space by detecting QR codes with known locations.

As a prototyping platform, I used a Raspberry Pi board with the Picam camera module. For QR codes, there is a library available to detect them, called zbar. The problem with zbar is that its performance is not good enough to use in robotics. On a Raspberry Pi 2B, it can scan the camera image about three times per second, which is too slow.

Therefore, I explored alternatives to QR codes. I created a new type of symbol that is simpler than a QR code and can be detected faster. The symbol detection was done using Python and the OpenCV library. OpenCV provides tools for computer vision, such as colorspace conversion or contour detection algorithms. These codes can be detected at about 20fps on the Raspberry Pi.

Due to the interpretation layer of Python, it’s not perfect for projects with critical performance. To increase performance even more, the code could be rewritten in C++, which I decided not to do for this prototype. However the performance was improved by operating on large arrays using Numpy rather than standard Python.

For even faster recognition, I designed symbols that just consist of concentric circles. These can be detected faster, but they don’t carry any data. The robot can’t tell them apart.

Now that the Pi knows where the symbols are, a laser is pointed at them. The setup can move a red laser on two axes, using two stepper motors. The motors are connected with 3D-printed parts and driven by two Sparkfun EasyDriver boards that provide the correct voltage and current for the motors. The Raspberry Pi uses two GPIO pins for each motor to set the direction and trigger a step.

Together with a stand for the camera, the Raspberry Pi can see symbols on the table and move the laser dot around on the table. The camera is also used to detect where the laser is pointing. Another Python script takes the symbol position on the screen and computes the corresponding motor positions.

As another way of demonstrating the laser movement and symbol detection, I wrote a version of the game Pong that is played without a screen, on the table. The laser dot is the ball and the paddles are symbols that are moved around on the table.

Finally, here is an album that shows the steps that are used by the program to analyze an image:

[](//imgur.com/yO13U)

16×16 LED Matrix

Sat, 12 Sep 2015 00:00:00 +0000

This is a 16×16 RGB LED matrix, made of 256 WS2812B LEDs. It’s powered by a Raspberry Pi and can display images and animations. With a game controller attached, it can play games.

The pictures below show how I built the frame.

Painting the base plate

Drawing the scale

Ordered a custom picture frame

Assembled frame

Frame with glass front and baseplate

Making the grid

Completed grid

Testing the LEDs

Sticked the first line of LEDs

All strips sticked on

Soldering complete

Mounting the Raspberry Pi

Relieved that it actually lights up

Assembled back side

Testing with a PC power supply while waiting for the actual one to arrive

Complete

Hello!

Artwork by Eboy

Beach

Drop

Fire

Diamond Sword

Link

Mario

Pacman

Space Invader

Working clock

It’s a Raspberry Pi, so why not use the USB port for a gamepad?

Tetris

Snake

This project was inspired by Jeremy William’s Game Frame. My code can play all animations that work with the Game Frame. Find my source code, a parts list and build notes on github. I also published these pictures to imgur.

Work in progress: Location based online game

Sat, 12 Sep 2015 00:00:00 +0000

This is a game prototype I’m currently working on. The game is played online, on a real world map and the location of the player is also the location ingame, just like in Ingress.

I know that making an online game like this is an ambitious goal and it will probably never be finished, but this prototype is a good way for me to test the game mechanic and see what works and what doesn’t.

The prototype is a web app and doesn’t yet use the player’s location, meaning they can interact anywhere on the map. If all works out, I’ll make a mobile client, which will be the actual game.

The game is about finding resources and mining them. I’m trying to make the resource locations meaningful and related to the real world. So I wrote an engine that procedurally distributes resources on the map, based on map data provided by Open Street Map. This allows me to define rules like “Resource 1 can be found at water fountains near public parks”, or “Resource 2 can be found only in forests and only near paths". Any information that is in OpenStreetMaps can be used to create rules. Here is an example:

The green resources are distributed randomly along streets, the pink ones are around monuments, orange resources can be found near shops. The goal of this is to encourage players to think about what they can do at a certain place without looking at the map. Also, different strategies will emerge in rural and urban areas and some places will be strategically more important than others. This is a feature that Ingress lacks, the Ingress gameplay differs very little across different locations.

There are four kinds of buildings that can be placed anywhere on the map.

In the actual game, players can only place buildings at their current location, and only if there isn’t another building already.

Buildings consume and output resources, which are transported via links. All buildings can be connected with links and the system will figure out the most effective way to use them.

A mine picks up resources from the ground.

A warehouse stores them.

There are ten tiers of resources and only the resources of the first tier can be aquired by mining. Higher tier resources are made by directing two different kinds of resources of the same tier into a factory and it will output a resource of a higher tier. This encourages players to build complex networks of mines, factories and warehouses, like this one:

And finally, a tower protects someones buildings. Buildings that are not in the range of a tower can be used by everyone. So you need to build a tower to keep people from pulling resources out of your factories and looting your warehouses. High-tier resources serve as explosives to take down other’s towers and use their production networks.

You can see that two of the mines are idle, because the warehouse has a limited input rate.

The goal of this mechanic is to encourage teamwork, reward efforts and provide some permanence to the players’ actions, as opposed to Ingress. In Ingress, it doesn’t really matter which portals of which place you conquer, since someone else will have taken them over by the next day. I’m trying to make a game that fixes some of the problems that Ingress has.

The screenshots in this article contain map data by OpenStreetMap (ODbL) and Dark Matter tiles by CartoDB (CC BY 3.0).

Game prototypes

Sat, 18 Jul 2015 00:00:00 +0000

I’d like to share two game prototypes I made a few years ago. The first one is based on Tetris:

It was written in C++ with bare-bones OpenGL. Once you press shift, the game enters a “fast mode”, where the down button takes a piece all the way down and if you don’t press it for three seconds, it will drop where it is. This is meant to be a fast-paced version of the original Tetris.

The second game is based on the Osmos mechanic, in a 3D space. If two cells touch each other, the bigger one absorbs the smaller one. The goal is to become the biggest cell by avoiding bigger cells and eating smaller ones. I wrote this in Java with the LWJGL library.

Here is a zip file with both games.

Procedural pixelart generator

Sat, 18 Jul 2015 00:00:00 +0000

I made a procedural pixelart generator that is inspired by the art style of the upcoming space adventure game No Man’s Sky.

Check it out and generate your own pixelart:

https://marian42.github.io/proceduralart/

The art generator is written in javascript and uses noise functions such as simplex noise to generate procedural landscapes and other features. Whenever the page is loaded, a new random seed is selected and an artwork is generated based on this seed. As a consequence, it is extremely unlikely that the same planet is shown twice.

Raspberry Pi powered fishtank

Fri, 03 Apr 2015 00:00:00 +0000

My fishtank is now internet-connected. It is run by a Raspberry Pi that can do three things: Feed the fish, switch the lights and take pictures.

To feed the fish, the Raspberry Pi sends commands to an automatic fishfeeder that I modified. It can empty any container in any order. This is achieved by an Arduino Pro Mini, two servos and a motor (more). The plate prevents hot air from flowing into the feeder. In an early iteration of the project, this made the food sticky and kept it from falling into the tank. Switching the lights is done using a remote light switch and 433Mhz transmitter. Finally, there is a webcam that is connected directly to the Raspberry Pi to take pictures.

The setup sits on top of my aquarium:

The Raspberry Pi runs a python / flask server that provides a web interface. If everything works, the web interface of my fishtank is available here:

http://fishtank.marian42.de/

The food stored in each container is remembered by the python script. This allows rules like “Feed the oldest food first” or “Prefer flake food”. The script will also try to minimize the rotation time.

The Raspberry Pi performs planned events that can be set up and monitored through the web interface.

To prevent overfeeding, the server remembers a “saturation value” for the fish. Whenever the fishfeeder adds food, the saturation value is increased (eg. by one for a typical one day portion). The saturation is gradually decreased by one per day, but doesn’t go below zero. A value below 1 means that the fish are hungy. Planned feed events are only automatically performed by the Raspberry Pi when the saturation is below 1. This mechanism makes sure that on average the fish get about one portion unit of food per day.

The web interface allows guests to feed the fish. This feature is only available if the fish are hungry and the lights are on. Guests can choose what kind of food to feed. Once the process is complete, a picture is taken and displayed on the website.

The pictures taken with the webcam can be used to verify that everything worked.

Whenever the fish are fed or a picture is taken, the picture is sent to my phone using the pushbullet API.

The server keeps a log of all events and errors that can also be viewed on the web interface.

All the source code for this project can be found on github.

Connecting my fish tank to the Internet of Things – Part 1: Hacking an automatic fish feeder

Mon, 30 Jun 2014 00:00:00 +0000

I bought a used automatic fish feeder from ebay. This device is completely mechanic and very old (older than 1989). It has 27 containers for fish food and a disc that does one rotation per day. By sticking pins into that disc one can trigger one or more feedings per day. A pin will rotate an outer ring with the containers by one unit. One container always faces down, emptying its contents. So if the outer ring has made one revolution, all containers have been emptied.

The goals of my modification were

Add an Arduino that can empty a container at any time
Let the ring move to any container without emptying all other ones along the way
Let the Arduino know the absolute position of ring

The advantage of this is the ability to store differend kinds of fish food and different portion sizes. This is possible since the feeder can not only empty the very next container, but any one.



Taking the old fish feeder apart	Removing the old clockwork	New gear routing

In order to hack the fish feeder, I removed the original power supply, most of the original clockwork and replaced the motor with a lower voltage, DC motor. The motor is driven using a Sparkfun Motor Driver board.

I added a servo that pushes a bar in the gap that used to empty all passing containers. When the correct container is above the gap, the servo moves down, releasing the containers’s content. I use an Arduino Pro Mini as microcontroller.

To make precise movements, I added a light sensor to one cog in the clockwork. I colored part of the cog black and put an LED underneath it. When the black part moves between the LED and the sensor, it senses less light. The microcontroller can use this to count the revolutions, mimicing the functionality of a stepper motor.

Another LED plus light sensor detects the position of the outer ring. A white marker on the black ring indicates the position of the first container, letting the microcontroller determine the absolute position of the ring.

I drilled a hole in the case of the fish feeder and put in an RGB-LED. It can display the status of the Arduino. For later use, I also connected another light sensor, facing into the fish tank to check whether the lights are on.

The Arduino Pro Mini and all the electronics fit nicely into the case of the original device. The Arduino will later be connected to a Raspberry Pi and communicate through a serial connection. The next entry will be about setting up the Raspberry Pi.

Arduino Source Code

How to build a Lego Portal Gun

Wed, 30 Apr 2014 00:00:00 +0000

Resources:

Parts: list, Rebrickable CSV, Bricklink XML
Model: LDR, 3DS
Build this MOC on Rebrickable

Instructions:

The model looks a bit better if this axle is shortened to 1.5 units. So if you are willing to sacrifice one axle and build bad lego, you can cut off the tip.

Android-Benachrichtigungen auf dem Schreibtisch

Sun, 16 Feb 2014 00:00:00 +0000

Das Ziel dieses Projekts ist, Benachrichtigungen von meinem Android-Handy automatisch auf einem LoL-Shield (Lots of LEDs) anzuzeigen. Dazu benutze ich einen Raspberry Pi, der sowieso schon auf meinem Schreibtisch steht, einen Arduino für das Schild und auf dem Handy Tasker, um auf Benachrichtigungen zu reagieren. Sobald eine Benachrichtigung erscheint, passiert folgendes:

Der Arduino-Sketch zeigt die entsprechenden Icons in 8 Graustufen an. Dazu habe ich ein Java-Programm geschrieben, das PNG-Grafiken mit 14×9 Pixeln in Arrays mit C++-Syntax umrechnet. Werden einzelne Icons per serieller Schnitstelle aktiviert, werden sie auf dem Shield angezeigt und bei mehreren periodisch durchgewechselt.

Der Arduino ist mit dem Raspberry Pi mit drei Kabeln verbunden: 5V, GND und TX. Der TX-Pin des Pi ist mit dem RX Pin des Arduino verbunden. Auf dem Pi läuft ein PHP-Script, das die Befehle an den Arduino weiterleitet. Die einzelnen Befehle für den Arduino können auch mit einem Webinterface ausgelöst werden.

Es gibt eine Demofunktion, die alle Symbole anzeigt:

Die Konfiguration mit Tasker gestaltet sich etwas schwieriger, da es keinen Trigger gibt, der bei jeder Benachrichtigung ausgelöst wird und gleichzeitig Rückschlüsse auf Anwendung zulässt, die die Benachrichtigung ausgelöst hat. Deshalb musste ich für jede unterstützte App (Twitter, Facebook, …) einen eigenen Task anlegen, insgesamt 12. Ebenso ist es nicht möglich, mit Tasker auf das Anklicken oder Deaktivieren von Benachrichtigungen zu reagieren. Deshalb werden beim Entsperren des Displays alle Symbole auf dem LoL-Shield deaktiviert und solche, die bei eingeschaltetem Handy-Display erschienen sind, verschwinden nach 40 Sekunden wieder.

Unabhängig von den Benachrichtigungen kann das LoL-Shield, wenn keine Icons angezeigt werden, eine Binäruhr anzeigen. Es gibt außerdem die Möglichkeit, Texte zum Arduino zu senden, die dann über das Display scrollen. Dies kann per Web-Interface geschehen oder mit der Benachrichtigungsfunktion kombiniert via Tasker weitere Informationen anzeigen, beispielsweise, wer eine Nachricht gesendet hat.

Download: Arduino Skript (benötigt LoL-Shield Bibliothek), PHP-Skript, (benötigt PHP Serial), Java Programm für Icons, Tasker-Skript

Aquarium Innensicht mit Gopro

Sat, 30 Nov 2013 00:00:00 +0000

Wegen meinem Quadrocopter habe ich eine Gopro, die ja auch wasserfest ist. Da war es naheliegend, die mal im Aquarium auszuprobieren. Bei dem Aquarium stand einiges an Arbeit an, da das Glasbecken undicht war, und alles komplett ausgeräumt werden muss, um das Becken auszutauschen.

Uni-Timer

Sat, 30 Nov 2013 00:00:00 +0000

Ich bin seit diesem Semester Student und in der Uni dauern Vorlesungen immer 90 Minuten, von “viertel nach” bis “viertel vor”. Da kam mir die Idee, dass man eine Uhr bräuchte, die nicht den Fortschritt der aktuellen Stunde, sondern den der aktuellen Vorlesung zeigt. Dazu habe ich schnell diese App zusammengecodet:

Die App zeigt die verbleibende Zeit oder den Fortschritt an und merkt sich den Namen der Veranstaltung und zeigt ihn nächste Woche zur selben Zeit wieder an. Und es gibt ein spezielles Feature für meinen Mathe-Prof, bei dem die Vorlesung nicht 90, sondern 95 Minuten dauert. Das Design ist dem der Stock-Uhr von Android 4 nachempfunden.

Download: Android-App, Source code

Quadrocopter

Sat, 26 Oct 2013 00:00:00 +0000

Ich habe mir dieses Jahr den Traum erfüllt, einen selbst zusammengestellten Quadrocopter zu bauen.

Investitionen

Für mich ist dieses Projekt bisher immer an zu hohen Kosten und mangelnden Informationen für Einsteiger gescheitert. Diese Probleme wurden zum Teil ausgeräumt durch die Erkenntnis, dass es deutlich günstigere Systeme als die von Mikrocopter gibt, als auch die Hilfe von der Community bei fpv-community.de, insbesondere dieser Einsteiger-Guide. Mein Ziel bei diesem Projekt ist, möglichst wenig zu investieren und einen Copter zu haben, der später problemlos aufgerüstet werden kann, z.B. mit einer Kamera oder einem FPV-System.

ArduCopter

Letztendlich benutze ich als Flight Control, also als “Gehirn” des Copters, einen ArduFlyer 2.5, der ein Nachbau des Open Source Projekts ArduPilot ist. Das Board ist baugleich und entsprechend kompatibel, bzw. bietet die gleichen Funktionen. Die Vorteile von ArduPilot sind die hohe Rechenleistung, die Sensoren (10 DOF) und die Kompatibilität zu verschiedener Hardware wie GPS, Kompass, Telemetrie, Kameraaufhängung.

Hobbyking und Teile

Einen Großteil der Teile habe ich bei Hobbyking bestellt, das besonders günstig ist, aber einen durchwachsenen Ruf hat. Bei mir sind keine der “bekannten” Probleme mit Hobbyking aufgetreten. Die Pakete waren nach einer Woche in Deutschland, wo sie allerdings recht lange beim Zoll bearbeitet wurden. Wer im Ausland bestellt, muss ab einem bestimmten Wert damit rechnen, 19% MwSt und 4,9% Zoll zusätzlich zu bezahlen.

Fernsteuerung

Als Fernsteuerung benutze ich die Turnigy 9X, die dadurch hervorsticht, dass sie nur etwa 10% einer typischen RC-Fernsteuerung kostet. Dies macht sich dadurch bemerkbar, dass einige Teile etwas “wackelig” wirken. Die Firmware ist ab Werk nicht zufriedenstellend, da sie Fehler enthält und bei jedem Tastendruck laut piept. Dank einer breiten Nutzerbasis gibt es allerdings auch alternative Firmwares, die sich mit etwas Löten und einem Arduino aufspielen lassen. Mit der er9x Firmware hat die Turnigy dann alle Features, die man sich wünschen kann. Außerdem habe ich eine Displaybeleuchtung und einen Akku nachgerüstet, was die 9X zu einer vollwertigen Fernsteuerung macht.

Technische Daten

Der Copter ist mit einem GPS-Modul, einem externen Kompass, einem Telemetriesystem zum Datenaustausch mit einer Bodenstation und LED-Streifen ausgestattet. Die Flugzeit beträgt ohne Kamera etwa 13 – 15 Minuten bei einem 3Ah Akku (12V, 3C, LiPo). Die ESCs sind mit der SimonK Firmware geflasht. Der Copter ist diagonal etwa 60cm lang und wiegt 1kg.

Fähigkeiten

Diese Stunts sind etwa das “Standardrepartoire” vom Ardupilot. Im Folgenden also eine Liste mit Dingen, die der Copter kann, allerdings nur die, die ich schon erfolgreich getestet habe:

Loitermodus: Der Copter behält automatisch die Höhe und Position bei
Return to Launch: Der Copter fliegt zum Startpunkt und landet. Dies ist auch das Verhalten, wenn die Verbindung zur Fernsteuerung abbricht.
Bodenstation: Mit einer Verbindung zu einem Handy / Laptop am Boden können Flugdaten abgerufen und Befehle gesendet werden, wie Wegpunkte, die der Copter anfliegt.
Follow Me: Der Copter folgt einer Person (bzw einem Handy) auf einer festgelegten Höhe

Kamera

Der typische Anwendungsfall von Multicoptern ist das Filmen oder Fotografieren aus der Luft. Mein Copter ist seit kurzem mit einer Gopro ausgestattet. Er wird bald ein Kamera-Gimbal bekommen, das die Neigungen des Copters während des Flugs ausgleicht.

Licht

Der Copter ist mit RGB-LED Streifen ausgestattet, die pro Arm ansteuerbar sind. Mit der Fernsteuerung kann durch verschiedene Modi durchgewechselt werden, wie “Polizei” oder einzelne Farben. Mit dieser Beleuchtung kann man bei Nacht schöne Effekte mit Langzeitbelichtung erzielen.

Material

Raspberry Pi Projekte

Sat, 19 Oct 2013 00:00:00 +0000

Was macht man eigentlich mit einem Raspberry Pi? Ich habe mir vor einem halben Jahr einen gekauft. Seitdem steht er hier auf meinem Schreibtisch und führt eine Reihe von Aufgaben aus. Hier eine Übersicht:

http-Server für einige Webseiten auf dem Pi und die Daten auf meiner externen Festplatte
Samba und FTP Server für die Daten auf der Festplatte

Wetterstation: Mit einigen Sensoren werden einmal pro Minute Wetterdaten aufgezeichnet

Webseite für die Wetterdaten
Über einen RF-Sender steuert der Pi einige Funksteckdosen in meinem Zimmer an (mit RF-Sender, ELRO-Funksteckdosen und rcswitch-pi)
Über einen IR-Sender steuert der Pi meine Stereoanlage (mit LIRC)

Für die Stereoanlage und die Steckdosen gibt es eine API und ein Web-Interface zum Steuern. Außerdem kann die App Tasker auf meinem Handy auf die API zugreifen. Es gibt z.B. einen Task für “Schreibtischlampe umschalten”. Auf dem Homescreen liegt ein Ordner mit Verknüpfungen für diese Tasks. Der eigentliche Sinn von Tasker ist aber, solche Tasks automatisiert auszuführen. Beispielsweise schaltet Tasker morgens, wenn der Wecker klingelt, auch gleich das Licht und das Radio an.
Der Pi verbindet ein LoL-Shield mit dem Internet, um Benachrichtigungen anzuzeigen.

Quadrocopter Lichtsystem

Tue, 15 Oct 2013 00:00:00 +0000

Dies ist ein Arduinoprojekt, das vier RGB-LED-Streifen an den vier Armen des Quadrocopters ansteuert. Das Ziel war, bei möglichst geringen Materialkosten möglichst viele Möglichkeiten bei der Beleuchtung des Quadrocopters zu haben.

Verwendete Teile:

1m RGB-LED Streifen, jeweils zwei Segmente a 3x RGB pro Arm, mit 12V
TLC5940 LED Driver zur Steuerung der LEDs
ATmega328p (der Microcontroller des Arduino Uno)
Bluetooth-Modul (muss im Flug nicht zwingend angebracht sein)
Kabel, zum Testen ein Breadboard

Der Arduino steuert über zwölf Kanäle jeweils den Rot-, Grün- und Blauanteil der vier LED-Streifen und kann so Farben mischen. Dazu wird der Farbwert (HSV) in RGB umgerechnet.

Als Eingabe dienen die Fernsteuerung des Quadrocopters, ein Bluetooth-Interface und der Status des Flightcontrollers. Die Fernsteuerung sendet über zwei Kanäle den Wert eines Potenziometers und den zweier Schalter (An/Aus, Modus). Abhängig vom Modus verändert der Drehregler unterschiedliche Paramter (siehe Video). Die gleichen Eingaben können auch über das Bluetooth-Interface gemacht werden. Außerdem können dort Parameter für eigene Animationen festgelegt werden, die dann auf dem ATmega gespeichert werden und im Flug als eigene Modi auftreten. Weiterhin gibt das Lichtsystem ein optisches Signal beim Starten, beim Sperren und Entsperren, bei fehlendem GPS-Signal und bei ausgeschalteter Fernsteuerung. Dazu liest der Arduino die Signale der APM-Pins für die Status-LEDs (A4 – A6).

Nachdem ich das Projekt einige Zeit mit einem Breadboard getestet habe, sind nun alle Komponenten auf ein Stück Lochbrett gelötet, mit Steckverbindungen, sodass ich die Teile auch für andere Projekte verwenden kann. Außerdem habe ich eine Programmierschnittstelle (rechts) für den Arduino angelötet.

Für das Android-Interface habe ich meine selbstgeschriebene Android-App verwendet, die es ermöglicht, Benutzeroberflächen für Arduino-Projekte zu erstellen. In der App werden die Farben der vier LED-Ausgänge in Echtzeit angezeigt. Weiterhin können der Modus und der entsprechende Wert verändert werden.

Mit diesen Parametern können eigene Animationen konfiguriert werden:

Farbwert
Offset der Farbwerte zwischen den vier Armen
Offset der Farbwerte für die einzelnen Arme
Geschwindigkeit, mit der sich der Farbwert verändert (0 -> keine Änderung)
Schritt, mit dem sich der Farbwert ändert (0 -> stufenlose Änderung)
Sättigung

Sättigung für die einzelnen Arme
Helligkeit für die einzelnen Arme
Gesamthelligkeit Sinus (Helligkeit pulsiert alle x Millisekunden)
Offset Helligkeit Sinus zwischen den vier Armen
Zeitabstand zwischen Pausen
Dauer der Pausen

In der App kann die aktuelle Konfiguration gespeichert werden.

Material: Arduino-Quellcode, Quadcopter Lightpainting Fotos

Quadcopter Lightpainting

Mon, 07 Oct 2013 00:00:00 +0000

Die Fotos wurden mit einem beleuchteten Quadrocopter, einem Stativ und 15 Sekunden Belichtung aufgenommen.

Bei diesen Fotos stimmte die Einstellung noch nicht, sodass sie zu dunkel sind:

Das letze Foto wurde mit einem kleineren Quadrocopter, einem Hubsan X4 aufgenommen, an den ich eine blaue LED mit Knopfzelle geklebt habe.

Raspberry Pi Wetterstation

Mon, 23 Sep 2013 00:00:00 +0000

Um die Daten, die meine Arduino-Wetterstation liefert, verfügbarer zu machen, habe ich mich entschieden, das Projekt jetzt mit einem Raspberry Pi weiterzuführen. Die Sensordaten werden wieder vom ILC-Board geliefert, das ich für den Schülerwettbewerb Intel Leibniz Challenge bekommen habe.

Dazu kommt ein weiterer Helligkeitssensor und ein weiterer Temperatursensor. Den Temperatursensor verwende ich, da er einen eigenen IC zum Kompensieren von Spannungsschwankungen etc. hat und so zuverlässiger ist als der Heißleiter des ILC-Boards, der über einen Spannungsteiler ausgelesen wird.

Für den Pi habe ich ein kurzes C++-Programm geschrieben, dass dank Wiring-Pi weitgehend den Arduinocode aus dem Vorgängerprojekt enthält. Dabei wird aus 5 Messungen für jeden Sensor der Mittelwert verwendet. Das Programm wird von der Kommandozeile aufgerufen und kann die Daten direkt ausgeben oder auf die SD-Karte speichern. Ein Skript führt das Programm alle 60 Sekunden aus und zeichnet so die Daten auf.

Der Pi ist ans Netzwerk angeschlossen und hostet eine Webseite, über die die Daten angezeigt werden können. Die Seite greift dabei auf die letzen gemessenen Daten zurück oder kann auf Wunsch auch eine eigene Messung anfertigen.

Bei den automatischen Messungen wird außerdem ein Datensatz an Xively gesendet (siehe auch), das eine API und eine Visualisierung der Daten bereitstellt.

Im Vergleich zum Vorgängerprojekt ist die Lösung mit dem Raspberry Pi zuverlässiger, da für die SD-Karte, die Uhrzeit etc. keine spezielle Hardware notwendig ist. Wie bei der Arduinowetterstation ist die Hardware noch nicht für den Betrieb im Freien ausgerüstet, wo die Wetterdaten aussagekräftiger wären.

BTduino documentation

Sat, 27 Jul 2013 00:00:00 +0000

The BTduino app sends data using the serial interface of a microcontroller and a bluetooth connection. The concept of the protocol is to send all data in text form. Each set of data consists of the name and the value, seperated by a colon. This way of communication is not the most efficient one, but it is easy to use and human readable. Example:

name:value

Special commands

Setup

requestsetup

The BTduino app will send this command if you open a new project or if you click Menu > Request setup. At this point the microcontroller program should request all events that it will be using. The example sketch for Arduino provides a method that is called when requestsetup is received.

Ping

ping

If the BTduino app receives this line, it will send exactly the same back to the microcontroller. This can be used to check if there is a working connection.

Remove event

remove:myevent

If the BTduino app receives this line, it will remove the specified event.

Parse

In your project in the BTduino app, click Menu > Parse and enter a command. This will have the exact same effect as if the app received this command via Bluetooth.

Events

BTduino features several event types that can be used in a project:

Value receiver

This event displays values that are sent from the microcontroller to the Android device. Send the name, a colon and the value as one line. The event is discovered by the app automatically. Example:

mysensor:42

The app provides a gauge, a plot and datalogging.

String receiver

This event displays text that is sent from the microcontroller to the Android device. Send the name, a colon and your text as one line. The event is discovered by the app automatically. Example:

log:ready

The app provides a datalogging feature.

Value sender

This event provides an interface to send numbers from the Android device to the microcontroller.

Send a value by typing it in the input field and hitting the send button or by using the slider. Touching the slider will automatically send a value to the microcontroller. The data will look like this:

myvaluesender:42

You can change the value of the input field and the position of the slider via Bluetooth. This is done again by sending the name and the value as in the example above.

The event can be added manually in the Android app or by sending this command via bluetooth:

requestint:myvaluesender

This command will as well make the Android app send the current value.

Set the min and max value of the slider by entering a value in the input field and then clicking the min or max value or by sending one of these commands via Bluetooth:

setminmyvaluesender:0
setmaxmyvaluesender:100

String sender

This event provides an interface to send text from the Android device to the microcontroller. Send text by typing it in the input field and hitting the send button. The data will look like this:

mystringsender:Hello World!

The event can be added manually in the Android app or by sending this command via bluetooth:

requeststring:mystringsender

This command will as well make the Android app send the current value.

Boolean sender

This event provides a switch that will send a boolean value to the microcontroller. Example:

mybooleansender:true;
mybooleansender:false;

You can change the state of the switch via Bluetooth. This is done again by sending the name and the value as in the example above. The app will recognize “true”/”false”, “1”/”0″ and “HIGH”/”LOW”. The event can be added manually in the Android app or by sending this command via bluetooth:

requestboolean:mybooleansender
requestbool:mybooleansender

This command will as well make the Android app send the current value.

Event sender

This event provides a button that will send a specific text to the microcontroller when clicked. It will send the word “event” and the name of the event:

event:myevent

Add it manually from the app or by sending this code:

registerevent:myevent

Color receiver

This event displays colors that are sent from the microcontroller to the Android device. Send the name, a colon and the value as one line. You can use the HTML format or send the color as an integer value. The event is discovered by the app automatically if you use the HTML color format. Example:

mycolorreceiver:#FF00FF
mycolorreceiver:16711935

Both lines represent the same color, because 16711935 equals 0xFF00FF. The color receiver provides a datalogging feature.

Color sender

This event provides an interface to send colors from the Android device to the microcontroller.

Click on the color area to choose a color and click the send button to send it. You can select between four color formats: Integer, Hex, HTML and RGB. Example:

mycolorsender:16711935
mycolorsender:FF00FF
mycolorsender:#FF00FF
mycolorsender:255,0,255

You can change the value of the color field via Bluetooth. This is done again by sending the name and the value as in the example above, using the HTML or integer format.

The event can be added manually in the Android app or by sending this command via bluetooth:

requestcolor:mycolorsender

This command will as well make the Android app send the current color.

Joystick

This event provides a virtual joystick on the Android device that will continuously send the position via bluetooth. It will send X and Y coordinates between -1.0 and 1.0 like this:

myjostick:0|0

The example sketch for Arduino shows how to use the data. The event can be added manually in the Android app or by sending this command via bluetooth:

requestjoystick:myjoystick

You can lock or invert an axis in the BTduino app. The app allow you use the accelerometer to control the joystick. Click the record button to record a path. It can be performed using the play button. After you have recorded a path, long-click the joystick and select “Save path” to add a path event. This allows you to record multiple paths.

The logfiles for value receiver, string receiver and color receiver are stored on the SD card int he folder /Bluetooth/<project name>/<eventname>.txt. The textfile contains one dataset per line, consisting of the timestamp in milliseconds and the value.

BTduino

Sat, 27 Jul 2013 00:00:00 +0000

BTduino is an Android app that lets you add a custom bluetooth interface to your Arduino project without any programming on the Android side.

Everything is better with bluetooth! Here are some examples:

Download

Android APK file Android 4.0 or higher is required.
Arduino example sketch
Arduino sketch without examples
Source code

To use this app, you need an Arduino or any microcontroller with a serial connection that you can program, a bluetooth module (about €10) and the parts for your project.

Getting started

The app is designed to be self-explaining. Send values by sending a name, a colon and the value, all in text form. Take a look at the example sketch for Arduino for a quick start. There is a step-by-step tutorial that shows you how to control an LED and a potentiometer via bluetooth. All capabilities of BTduino are explained in the documentation. If anything goes wrong, check out the troubleshooting page.

Beta

I have tested this only with my own setup, so consider this a beta release. If you find a bug or want to request a feature, please send me an email. Also, just out of curiosity, please let me know if you used this for a project.

How to add Bluetooth to your Arduino Project with BTduino

Wed, 24 Jul 2013 00:00:00 +0000

This tutorial will show you how to connect your Arduino project to an Android device using the BTduino app. You don’t need an extra Arduino library and you don’t need to code anything on the Android side.

Here is what you need:

an Android device running Android 4.0 or higher that supports bluetooth (most phones do)
an Arduino or any microcontroller that has a Serial connection
a bluetooth module for the Arduino (get one for about €10, for example from ebay)
the parts for your project

Connect the bluetooth module to your Arduino like this:

If your bluetooth module runs at 3.3V, you need an additional voltage divider. In this case, use three resistors with an equal value between 1kΩ and 10kΩ as shown above.

Then, set up your Arduino project. For this tutorial I will be using a potentiometer and an LED. Here is the setup:

Now install the BTduino app on your device:

Download it here or scan the QR code on the right.

The main concept of the BTduino app is to send data through the serial interface in text form. Every set of data contains the name of the event, a colon and the value:

name:value

To send the value of the potentiometer to the BTduino app, you just have to make the Arduino send something like “A0:4”. Here is the code:

Serial.begin(9600);

void loop() {
    Serial.println("A0:" + String(analogRead(0)));
    delay(200);
}

Now it’s time to start the app and establish a bluetooth connection. To do so, click on the connect button on the bottom right of the screen. Enable bluetooth if prompted. Your bluetooth module should show up in the list. If you are asked for a pin, try 0000 or 1234.

You can show the raw text data that is sent by the Arduino using the bluetooth console. However, the point of this app is to interpret the received data. Click “Exchange values” and a screen will show up that displays the events of your bluetooth project. The event that was named “A0” by the above Arduino code will show up automatically. You can display a gauge or a plot of the data by clicking the icons next to the value display. Play around with the potentiometer to see how the value changes.

To add an event for the LED, click the plus icon in the BTduino app. Enter “led” for the name and select the “Send boolean” event. Every time you click the button on the new event, the app will send either “led:false” or “led:true” to the Arduino

Receiving data at the Arduino is a bit more complicated than sending because you will have to read and interpret the serial data. So a good point to start is this example sketch that provides a base for BTduino projects.

Open a new Arduino sketch and paste the content of the example sketch. To use the LED, declare a variable for the pin and set the pin mode in the setup function:

int ledpin = 13;
...
pinMode(ledpin,OUTPUT);

Every time a full line of text is received, the function receiveline will be called. After the comment Check events you have acces to the string variables name and value. To switch the LED, add this code:

if (name == "led") {
    if (value == "true") digitalWrite(ledpin,HIGH);
    if (value == "false") digitalWrite(ledpin,LOW);
}

This is one of the examples provided by the sketch so you just need to remove the comment. Upload the sketch and you should be able to turn the LED on and of from your Android device!

That’s it for the tutorial. You can take a closer look at the example sketch to see what else this app is capable of. Also, every event that you can add to your project in BTduino provides an example if you long-click it and select “About”. For more information, take a look at the events documentation. There is a troubleshooting page for frequent problems. Please send me an email if you have any problems, if you want to report a bug or request a feature or if you built a project with BTduino.

Also, you might want to take a look at this guide to change some of the bluetooth module’s settings. I changed the baud rate of my module to 57600. At the standard rate of 9600 it takes about 1ms to send one character which can slow down a time-critical project.

Arduino-Wetterstation mit Bluetooth, Datalogging und Android-App

Tue, 09 Jul 2013 00:00:00 +0000

Die Wetterdaten für die Wetterstation werden von einem Board gesammelt, das ich für einen Schülerwettbewerb (ILC) bekommen und zusammengesetzt habe. Auf dem Board rechnet ein ATxmega128A3U. Temperatur, Luftfeuchte, Luftdruck, Helligkeit, Regenmenge und Windgeschwindigkeit werden bestimmt. Die letzen beiden Werte werden durch mechanische Aufbauten ermittelt, an denen ein Magnet einen Hallsensor passiert.

Auf dem Breadboard befindet sich ein ATmega328p, auf dem Arduino-Code läuft. Das ILC-Board sammelt und sendet die sechs Wetterdaten auf Anfrage an den Arduino. Aus Zeitgründen habe ich dafür ein eigenes Protokoll implementiert, diese Aufgabe könnte eleganter und besser mit I²C o.ä. umgesetzt werden.

Der Arduino ermittelt über eine angeschlossene DCF77-Antenne die aktuelle Uhrzeit, wie in einer Funkuhr. Mithilfe eines SD-Moduls wird auf einer SD-Karte eine Textdatei angelegt, in der die Wetterdaten gespeichert werden. Die Daten werden alle vier Sekunden aufgenommen und gespeichert, mit einer Zeile pro Datensatz. Dabei verwende ich das CSV-Format, sodass die Daten später einfach ausgewertet werden.

An die serielle Schnittstelle des Arduinos ist ein Bluetooth-Modul angeschlossen. Eine selbst entwickelte Android-App empfängt die Wetterdaten und zeigt sie an. Dazu werden die Daten in Textform über die serielle Verbindung gesendet, im Format Name-Doppelpunkt-Wert, also z.B. “Temperatur:25.41”. Neben den Wetterdaten werden in der App auch die Uhrzeit des DCF-Moduls und Informationen zur Stromversorgung angezeigt.

Als Stromversorgung dient entweder ein Akku von einem Modellauto (7,2V, 3Ah), der auf dem Board auf 5V reguliert wird, oder ein Handyladegerät, das über den Micro-USB-Port an das ILC-Board angeschlossen wird. Die Widerstände auf dem Breadboard dienen dazu, zwischen dem Arduino mit 5V und dem ILC-Board (bzw. dem DCF77-Modul) mit 3,3V zu kommunizieren.

Der eine Knopf setzt den Arduino zurück und der andere schaltet das Bluetooth-Modul an oder aus. Die rote LED leuchtet auf, wenn ein Fehler auftritt, wie zum Beispiel ein fehlerhaftes Zeitsignal oder eine fehlende SD-Karte. Die rechte grüne LED leuchtet immer und die linke visualisiert das DCF-Signal, sodass man erkennen kann, ob die Antenne richtig ausgerichtet ist.

Der Hauptnachteil von meinem Projekt ist, dass es nicht wetterfest ist und somit nicht ohne zusätzliches Gehäuse draußen aufgestellt werden kann. Die Wetterstation bringt den ATmega an seine Grenzen was Programm- und Arbeitsspeicher betrifft. Für weitere Features wie eine aktive Steuerung über Bluetooth wäre also z.B. ein Arduino Mega notwendig. Ein weiteres Problem ist die Zuverlässigkeit des DCF77-Moduls, das drinnen oft kein Signal empfängt. In Zukunft plane ich, ein ähnliches Projekt mit einem Raspberry Pi umzusetzen, der dann alle Aufgaben außer dem Sammeln der Sensordaten übernimmt. Die Daten könnten so über das Internet bereitgestellt werden und die Beschränkung durch die Bluetooth-Reichweite entfällt.

Ich habe mit dem Projekt bei der Intel Leibniz Challenge 2013 teilgenommen, bin aber nicht in die Endauswahl gekommen. Für diesen Wettbewerb habe ich auch eine GUI entwickelt, mit der die Daten über eine USB-Verbindung am Computer angezeigt werdenkönnen.

Downloads: Arduino-Sketch, Arduino-Sketch für den ATtiny, Android-App (mindestens Android 4.0), Fritzing-Schaltplan.