Doing random things over at http://musteat.org
227 stories
·
4 followers

Trace Together Token: Teardown and Design Overview

1 Share

On 19 June, GovTech Singapore invited four members of the community to come and inspect their new TraceTogether Token. This token removes the need to carry a phone at all times, and is designed to help both those who do not have a smart device capable of running TraceTogether well, including those using older Android devices, non-smartphones, and iOS users. I was among the group, which also consisted of Roland Turner, Harish Pillay, and Andrew "bunnie" Huang, who were given the opportunity to see the first public revision of hardware. In this post I will discuss the goal of the token, give some overview of the hardware, compare it with the app version of TraceTogether, and comment on the protocol changes.

Goal of the TraceTogether Token

The Trace Together Token is a dedicated hardware device that makes it easier to inform people if they may have come in prolonged contact with a person who subsequently was diagnosed with COVID-19. This is its sole purpose.

It is a hardware implementation of the app that GovTech previously developed, and has been installed over half a million times. The TraceTogether Token builds on the app and simplifies its usage: Throw it in a handbag or attach it to a keychain and forget about it while it does its thing.

Comparison With Phone Apps

I won't do an in-depth analysis of the Trace Together app. You can read an independent analysis that Frank Liauw put together to learn more. I worry more about the amount of spying that other popular apps on my phone do. For example, every banking, taxi booking app, and food delivery service on my phone has uploaded some amount of data to Facebook, Google, and a company called AppFlyer.

TraceTogether uses a protocol known as BlueTrace, and there are several problems with the protocol that make it challenging to work with.

First, antenna designs vary. As part of the Bluetooth spec, devices can report the amount of power they are currently using to broadcast:

alt
From the Supplement to Bluetooth Core Specification, Part A

BlueTrace includes this information in the advertising beacon, but what does the number mean? Intuitively it will tell you how far away a device is, because you can correlate the strength of the received signal with the broadcast power: If they said they were loudly broadcasting but you received a weak signal, they must be far away. However, antenna designs vary, and just like with humans, one phone's "loud" is another phone's "whisper".

There's also the issue of charging. Phones must be charged at least daily. With mobile payment becoming more common, people are incentivized to keep their devices charged and running, however it's still very common to be hobbled by a phone simply running out of battery.

On Android, many device manufacturers are very aggressive when it comes to terminating background processes. TraceTogether necessarily must always run in the background, but your phone might not realize that and could terminate the process anyway. Anyone who has found themselves unable to receive WhatsApp messages without opening up the app will have experienced something similar. Additionally, many older Android devices are not compatible with the approach of TraceTogether, where a device can act as a Bluetooth Central (i.e. "host") or a Peripheral (i.e. "device").

As an aside, it was refreshing to hear Minister Vivian Balakrishnan using the words "Peripheral" and "Central" when discussing issues pertaining to older devices. These are the technical terms for the roles Bluetooth devices play, and he correctly pointed out that devices that do not support Bluetooth Low Energy (BLE) are incompatible with Bluetooth-based contact tracing.

Then there's the iOS problem. Apple does not let apps use Bluetooth in the background, so users must always run TraceTogether. The easy answer here is to use the Apple-sanctioned tracing protocol, however this protocol is not compatible with BlueTrace and makes very different assumptions about how contact tracing should work. It also assumes that everyone has a modern device, which leaves out a significant portion of the population that does not have the latest hardware.

Hardware Overview

The hardware isn't ready yet, and we aren't yet allowed to share photos of the device because supplier contracts are still being worked out. Still, we could identify the major components well enough that I put together this block diagram:

alt
Block Diagram of the Trace Together Token

There are several interesting parts to note about this diagram:

  • There is no battery charger – it's designed to run for several months on a single battery.
  • There is a realtime clock with its own battery meaning time must be important in the new protocol
  • They use a powered antenna to improve performance
  • The entire system must be extremely low power

The last point means it is unlikely that they hid a GPS tracker, WiFi radio, or cellular modem in this device. The battery is a small coin cell, which would only last a few hours if it were receiving GPS or communicating via WiFi. Additionally, there are no additional sensors such as an accelerometer, pressure sensor, or microphone.

All of the major ICs had obfuscated markings so we couldn't identify part numbers. However, the block diagram I sketched above happens to look very similar to the block diagram for Simmel, which is an open-source tracking token put together by Bunnie and myself:

alt
Block diagram of Simmel

There are a few notable differences:

  • Simmel uses a PCB antenna because it was a proof-of-concept device, so we were willing to accept reduced range.
  • There was an experiment in using Near Ultrasound as an alternative to Bluetooth for contact tracing, under the theory that it would be lower power. Ultrasound turns out to be too directional, which is why bats can use it for echolocation.
  • TraceTogether uses a separate realtime clock, whereas Simmel currently relies on a stupendous hack to save time in case of a crash, meaning it is less accurate.
  • Simmel runs CircutPython in an effort to make early development easier, with the intention of rewriting it in a lower level language later on. The TraceTogether Token presumably runs its own stack.

Despite these minor differences, there are many similarities:

  • Both run on a non-rechargeable battery
  • Both use a voltage regulator to stabilize the battery voltage
  • Both have an external flash memory for storing contacts
  • Both are designed to last for months without user interaction, and be forgotten about until they are needed
  • Both rely on Bluetooth – specifically Bluetooth LE – for interaction
  • Neither contains any additional sensors, since those would add cost and power consumption

Therefore, while the exact hardware details of the Trace Together Token are still obfuscated, I can safely say that it is conceptually extremely similar to Simmel. Any questions you have about the hardware approach GovTech is taking can be answered by looking at the Simmel hardware repository.

BlueTrace Protocol Changes

One of the challenges we ran into when developing Simmel was the power budget. The nRF52833 part we used requires a lot of power to listen to Bluetooth:

alt
The nRF52833 reference manual on power consumption (Receiving)

BlueTrace advises that a device listen 20% of the time, which means it's receiving 20% of the time. In testing, we were observing about 5.9 mA current draw when receiving data, compared to 0.012 mA when idle. Furthermore, BlueTrace recommends that a device must transmit about 90% of the time, which is unfortunate because transmission has similar numbers:

alt
The nRF52833 reference manual on power consumption (Transmitting)

For a mobile phone these numbers are tiny, but when running off of a pair of AAA batteries, a continuous drain of 2 mA means a 1200 mAh AAA battery will be drained in 1200 mAh / 24 hours / 2 mA = 25 days.

Part of the problem involves the BlueTrace approach, which follows a traditional Peripheral/Central approach: A Central advertises a GATT Service of "BlueTrace", allowing another device to connect and read the current temporary token / deposit its own temporary token. This is required in order to make a two-way connection in BLE, so it is an extremely common approach. Additionally, ensuring the connection is two-way also ensures that the devices are close enough to matter from an epidemiological perspective.

GovTech must have also run into these issues, because the BlueTrace protocol is being modified for use with the Trace Together Token. Instead of forming a two-way connection, devices now simply broadcast their temporary tokens. The interval for broadcast is much longer, and the scanning interval is much shorter, meaning the device can spend most of its time in a low-power suspend state.

Storage

Simmel has 2 MB of flash memory. Each BlueTrace Temporary Token payload is 160 bytes. That means Simmel can store (2^21) bytes of flash / 160 bytes per payload / 21 days =  624 records per day. We weren't sure how many daily records we could expect, and the number surely was very low during the Circuit Breaker. GovTech must have decided that this was too large to store on an embedded device, so for the Trace Together Token they reduced the size of the critical part of the payload. In the new protocol, this payload is much smaller, although we don't yet have exact numbers.

Finally, there's the issue of temporary tokens. In the current Trace Together app, a batch of tokens is downloaded from the Ministry of Health every few days. With Simmel, we assumed we could simply download a few months' worth of tokens and store them on the flash memory, consuming them as time passed. GovTech took a different approach with the Trace Together Token, and instead derive the temporary token from a unique ID hashed together with the current time. This approach is similar to how the European DP3T protocol works.

As a result of all of these changes, many of the challenges we faced when designing Simmel are avoided with the Trace Together Token: They can reduce power by spending less time transmitting and receiving; they don't have to use so much storage to keep track of interactions; and they don't need to store several months' worth of temporary tokens. Overall these changes are exactly what are needed for an implementation of the protocol for use in a hardware token.

Conclusions

The approach taken by GovTech when designing the Trace Together Token hardware is sound. The device accomplishes the goals it set out to, while preserving the privacy of the owner. Like the Trace Together app, the Trace Together Token cannot be used to identify the owner merely by looking at Bluetooth broadcasts – the only entity that can correlate logged data to a human is the Ministry of Health.

During this session we didn't have access to the software, because as a tracing beacon there's not much of a user interface beyond a blinking LED. Also, they weren't ready to let us attach debug probes, meaning we can't draw any conclusions about the software itself. However, given the PCB design and the system's power requirements there isn't much they could hide.

Overall I'm pleased with the direction they are going in with the Trace Together Token, and look forward to getting one of my very own.

Read the whole story
smarkwell
18 days ago
reply
Share this story
Delete

On Contact Tracing and Hardware Tokens

3 Shares

Early in the COVID-19 pandemic, I was tapped by the European Commission to develop a privacy-protecting contact tracing token, which you can read more about at the Simmel project home page. And very recently, Singapore has announced the deployment of a TraceTogether token. As part of their launch, I was invited to participate in a review of their solution. The urgency of COVID-19 and the essential challenges of building supply chains means we are now in the position of bolting wheels on a plane as it rolls down the runway. As with many issues involving privacy and technology, this is a complicated and nuanced situation that cannot be easily digested into a series of tweets. Thus, over the coming weeks I hope to offer you my insights in the form of short essays, which I will post here.

Since I was only able to spend an hour with the TraceTogether token so far, I’ll spend most of this essay setting up the background I’ll be using to evaluate the token.

Contact Tracing

The basic idea behind contact tracing is simple: if you get sick, identify your close contacts, and test them to see if they are also sick. If you do this fast enough, you can contain COVID-19, and most of society continues to function as normal.

However, from an implementation standpoint, there are some subtleties that I struggled to wrap my head around. Dr. Vivian Balakrishnan, the Minister-in-charge of the Smart Nation Initiative, briefly stated at our meeting on Friday that the Apple/Google Exposure Notification system did not reveal the “graph”. In order to help myself understand the epidemiological significance of extracting the contact graph, I drew some diagrams to illustrate contact tracing scenarios.

Let’s start by looking at a very simple contact tracing scenario.

In the diagram above, two individuals are shown, Person 1 and Person 2. We start Day 1 with Person 1 already infectious yet only mildly symptomatic. Person 1 comes in contact with Person 2 around mid-day. Person 2 then incubates the virus for a day, and becomes infectious late on Day 2. Person 2 may not have any symptoms at this time. At some future date, Person 2 infects two more people. In this simple example, it is easy to see that if we can isolate Person 2 early enough, we could prevent at least two future exposures to the virus.

Now let’s take a look at a more complicated COVID-19 spread scenario with no contact tracing. Let’s continue to assume Person 1 is a carrier with mild to no symptoms but is infectious: a so-called “super spreader”.

The above graphic depicts the timelines of 8 people over a span of five days with no contact tracing. Person 1 is ultimately responsible for the infection of several people over a period of a few days. Observe that the incubation periods are not identical for every individual; it will take a different amount of time for every person to incubate the virus and become infectious. Furthermore, the onset of symptoms is not strongly correlated with infectiousness.

Now let’s add contact tracing to this graph.

The graphic above illustrates the same scenario as before, but with the “platonic ideal” of contact tracing and isolation. In this case, Person 4 shows symptoms, seeks testing, and is confirmed positive early on Day 4; their contacts are isolated, and dozens of colleagues and friends are spared from future infection. Significantly, digging through the graph of contacts also allows one to discover a shared contact of Person 4 and Person 2, thus revealing that Person 1 is the originating asymptomatic carrier.

There is a subtle distinction between “contact tracing” and “contact notification”. Apple/Google’s “Exposure Notification” system only perform notifications to the immediate contacts of an infected person. The significance of this subtlety is hinted by the fact that the protocol was originally named a “Privacy Preserving Contact Tracing Protocol”, but renamed to the more accurate description of “Exposure Notification” in late April.

To better understand the limitations of exposure notification, let’s consider the same scenario as above, but instead of tracing out the entire graph, we only notify the immediate contacts of the first person to show definite symptoms – that is, Person 4.

With exposure notification, carriers with mild to no symptoms such as Person 1 would get misleading notifications that they were in contact with a person who tested positive for COVID-19, when in fact, it was actually the case that Person 1 gave COVID-19 to Person 4. In this case, Person 1 – who feels fine but is actually infectious – will continue about their daily life, except for the curiosity that everyone around them seems to be testing positive for COVID-19. As a result, some continued infections are unavoidable. Furthermore, Person 2 is a hidden node from Person 4, as Person 2 is not within Person 4’s set of immediate notification contacts.

In a nutshell, Exposure Notification alone cannot determine causality of an infection. A full contact “graph”, on the other hand, can discover carriers with mild to no symptoms. Furthermore, it has been well-established that a significant fraction of COVID-19 infections show mild or no symptoms for extended periods of time – these are not “rare” events. These individuals are infectious but are well enough to walk briskly through crowded metro stations and eat at hawker stalls. Thus, in the “local context” of Singapore, asymptomatic carriers can seed dozens of clusters in a matter of days if not hours, unlike less dense countries like the US, where infectious individuals may come in contact with only a handful of people on any given day.

The inability to quickly identify and isolate mildly symptomatic super-spreaders motivates the development of the local TraceTogether solution, which unlocks the potential for “full graph” contact tracing.

On Privacy and Contact Tracing

Of course, the privacy implications of full-graph contact tracing are profound. Also profound are the potential health risks and loss of life absent full-graph contact tracing. There’s also a proven solution for containing COVID-19 that involves no sacrifice of privacy: an extended Circuit-Breaker style lockdown. Of course, this comes at the price of the economy.

Of the three elements of privacy, health, or economy, it seems we can only pick two. There is a separate and important debate about which two we should prioritize, but that is beyond the context of this essay. For the purpose of this discussion, let’s assume contact tracing will be implemented. In this case, it is incumbent upon technologists like us to try and come up with a compromise that can mitigate the privacy impact while facilitating public policy.

Back in early April, Sean ‘xobs’ Cross and I were contacted by the European Commission’s NGI program via NLnet to propose a privacy-protecting contact tracing hardware token. The resulting proposal is called “Simmel”. While not perfect, the salient privacy features of Simmel include:

  1. Strong isolation of user data. By disallowing sensor fusion with the smartphone, there is zero risk of GPS or other geolocation data being leaked. It is also much harder to do metadata-based attacks against user privacy.
  2. Citizens are firmly in control. Users are the physical keeper of their contact data; no third-party servers are involved, until they volunteer their data to an authority by surrendering the physical token. This means in an extreme case, a user has the option of physically destroying their token to erase their history.
  3. Citizens can temporarily opt-out. By simply twisting the cap of the token, users can power the token down at any time, thus creating a gap in their trace data (note: this feature is not present on the first prototypes).
  4. Randomized broadcast data. This is a protocol-level feature which we recommend to defeat the ability for third parties (perhaps an advertising agency or a hostile government) from piggy backing on the protocol to aggregate user locations for commercial or strategic benefit.

Why a Hardware Token?

But why a hardware token? Isn’t an app just better in so many ways?

At our session on Friday, the TraceTogether token team stated that Singapore needs hardware tokens to better serve two groups: the underprivileged, and iPhone users. The underprivileged can’t afford to buy a smartphone; and iPhone users can only run Apple-approved protocols, such as their Exposure Notification service (which does not enable full contact tracing). In other words, iPhone users, like the underprivileged, also don’t own a smartphone; rather, they’ve bought a phone that can only be used for Apple-sanctioned activities.

Our Simmel proposal makes it clear that I’m a fan of a hardware token, but for reasons of privacy. It turns out that apps, and smartphones in general, are bad for user privacy. If you genuinely care about privacy, you would leave your smartphone at home. The table below helps to illustrate the point. A red X indicates a known plausible infraction of privacy for a given device scenario.

The tracing token (as proposed by Singapore) can reveal your location and identity to the government. Nominally, this happens at the point where you surrender your token to the health authorities. However, in theory, the government could deploy tens of thousands of TraceTogether receivers around the island to record the movement of your token in real-time. While this is problematic, it’s relevant to compare this against your smartphone, which typically broadcasts a range of unique, unencrypted IDs, ranging from the IMEI to the wifi MAC address. Because the smartphone’s identifiers are not anonymized by default, they are potentially usable by anyone – not just the government – to identify you and your approximate location. Thus, for better or for worse, the design of the TraceTogether token does not meaningfully change the status quo as far as “big infrastructure” attacks on individual privacy.

Significantly, the tracing token employs an anonymization scheme for the broadcast IDs, so it should not be able to reveal anything about your location or identity to third parties – only to the government. Contrast this to the SafeEntry ID card scanner, where you hand over your ID card to staff at SafeEntry kiosks. This is an arguably less secure solution, as the staff member has an opportunity to read your private details (which includes your home address) while scanning your ID card, hence the boxes are red under “location” and “identity”.

Going back to the smartphone, “typical apps” – say, Facebook, Pokemon Go, Grab, TikTok, Maps – are often installed with most permissions enabled. Such a phone actively and routinely discloses your location, media, phone calls, microphones, contacts, and NFC (used for contactless payment and content beaming) data to a wide variety of providers. Although each provider claims to “anonymize” your data, it has been well-established that so much data is being published that it is virtually a push of a button to de-anonymize that data. Furthermore, your data is subject to surveillance by several other governments, thanks to the broad power of governments around the world to lawfully extract data from local service providers. This is not to mention the ever-present risk of malicious actors, exploits, or deceptive UI techniques to convince, dupe, or coerce you to disclose your data.

Let’s say you’re quite paranoid, and you cleverly put your iPhone into airplane mode most of the time. Nothing to worry about, right? Wrong. For example, in airplane mode, the iPhone still runs its GPS receiver and NFC. An independent analysis I’ve made of the iPhone also reveals occasional, unexplained blips on the wifi interface.

To summarize, here are the core arguments for why a hardware token offers stronger privacy protections than an app:

No Sensor Fusion

The data revealed by a hardware token is strongly limited by its inability to perform “sensor fusion” with a smartphone-like sensor suite. And even though I was only able to spend an hour with the device, I can say with a high degree of confidence that the TraceTogether token has little to no capability beyond the requisite BLE radio. Why do I say this? Because physics and economics:

Physics: more radios and sensors would draw more power. Ever notice how your phone’s battery life is shorter if location services are on? If the token is to last several months on such a tiny battery, there simply is not enough power available to operate much more than the advertised BLE functions.
Economics: more electronics means more cost. The publicly disclosed tender offering places a cap on the value of parts at S$20, and it essentially has to be less than that because the producer must also bear their development cost out of the tender. There is little room for extraneous sensors or radios within that economic envelope.

Above: the battery used in the TraceTogether token. It has a capacity of 1000mAh. The battery in your smartphone has a capacity of around 3x of this, and requires daily charging.

The economics argument is weaker than the physics argument, because the government could always prepare a limited number of “special” tokens to track select individuals at an arbitrary cost. However, the physics argument still stands – no amount of money invested by the government can break the laws of physics. If Singapore could develop a mass-manufacturable battery that can power a smartphone sensor suite for months in that form factor – well, let’s just say the world would be a very different place.

Citizen Hegemony over Contact History

Assuming that the final TraceTogether token doesn’t provide a method to repurpose the Bluetooth Low-Energy (BLE) radio for data readout (and this is something we hope to confirm in a future hackathon), citizens have absolute hegemony over their contact history data, at least until they surrender it in a contact tracing event.

As a result the government is, perhaps inadvertently, empowering citizens to rebel against the TraceTogether system: one can always crush their token and “opt-out” of the system (but please remove the battery first, otherwise you may burn down your flat). Or perhaps more subtly, you can “forget your token at home”, or carry it in a metallized pouch to block its signal. The physical embodiment of the token also means that once the COVID-19 pandemic is under control, destroying the token definitively destroys the data within it – unlike an app, where too often uninstalling the app simply means an icon is removed from your screen, but some data is still retained as a file somewhere on the device.

In other words, a physical token means that an earnest conversation about privacy can continue in parallel with the collection of contact tracing data. So even if you are not sure about the benefit of TraceTogether today, carrying the token allows you to defer the final decision of whether to trust the government until the point where you are requested to surrender your token for contact trace extraction.

If the government gets caught scattering BLE receivers around the island, or an errant token is found containing suspicious circuitry, the government stands to lose not just the trust of the people, but also access to full-graph contact tracing as citizens and residents dispose of tokens en masse. This restores a certain balance of power, where the government can and will be held accountable to its social contract, even as we amass contact tracing data together as a whole.

Next Steps

When I was tapped to independently review the TraceTogether token, I told the government that I would hold no punches – and surprisingly, they still invited me to the introductory session last Friday.

This essay framed the context I will use to evaluate the token. “Exposure notification” is not sufficient to isolate mildly symptomatic carriers of COVID-19, whereas “full graph” contact tracing may be able to make some headway against this problem. The good news is that the introduction of a physically embodied hardware token presents a safer opportunity to continue the debate on privacy while simultaneously improving the collection of contact tracing data. Ultimately, deployment of a hardware token system relies upon the compliance of citizens, and thus it is up to our government to maintain or earn our trust to manage our nation’s best interests throughout this pandemic.

I look forward to future hackathons where we can really dig into what’s running inside the TraceTogether token. Until then, stay safe, stay home when you can, and when you must go outside, wear your mask!

PS: You should also check out Sean ‘xobs’ Cross’ teardown of the TraceTogether token!

Read the whole story
smarkwell
18 days ago
reply
denubis
19 days ago
reply
Share this story
Delete

Black Tech for Black Lives: Portland founders among those imploring tech to take a stand

1 Share

Among the names of 150 Black tech leaders imploring technology companies and startups to take a stand against systemic racism — through a newly launched effort called Black Tech for Black Lives — a couple of names immediately stood out. That’s because they were Portland founders Stephen Green, founder of PitchBlack, and Lindsey Murphy, founder of The Fab Lab.

We are a collective of Black tech entrepreneurs, investors, creatives, changemakers, and workers, united to use our social, political, and economic capital for the advancement of our communities. We commit to acting in solidarity with those leaders working to create a more just world.

Tech is complicit. We as Black people in tech have a unique position and opportunity to respond to violence against Black people’s bodies. While we’re proximate to the pain, we largely avoid its most brutal physical outcomes. But we, too, feel the blows. We carry the scars on our psyches and hearts as our voices go largely unheard in the workplace and beyond.

As Portland writer Taylor Hatmaker highlights in TechCrunch:

The effort, called “Black Tech for Black Lives,” pulls together a set of specific, actionable commitments intended to “support frontline leaders working to create a more just world.” The pledge is designed to elevate Bay Area community leaders working in tech’s epicenter on specific policy goals regarding issues like policing reform, local elections and by hiring and supporting more Black talent in tech.

To read more or to join as a cosigner or an ally, visit Black Tech for Black Lives.

Read the whole story
smarkwell
33 days ago
reply
Share this story
Delete

I started this project in 2015 with cute animals doing funny...

2 Shares


I started this project in 2015 with cute animals doing funny sounding new economy jobs. As it grew it added visionaries, disruptors and hackers. Then opportunists and con men. Violent billionaire dictator is not a random result here; it’s a parallel outcome of move-fast-and-break-things culture — an idea that only works for people who are so privileged and insulated that they have little to lose by plowing ahead without thinking about the consequences. 

Stop idolizing a few jackasses that won the VC lottery and listen to marginalized people instead. Have the hard conversations with our kids that so many of our parents avoided. 

Read the whole story
smarkwell
38 days ago
reply
Share this story
Delete

Side Scroller

2 Shares

I’ve never met Ian Albert, but years ago he painstakingly scraped and pasted together a set of maps and backgrounds from a various oldschool games, an effort that’s helped me in a bunch of odd little ways over the years and for which I’m grateful. Of particular interest today are the original Super Mario Brothers maps; for the sake of this exercise, let’s start with world 1, level 1.

ImageMagick and FFMpeg are a pair of “classically-Linux” command-line tools, in terms of how insanely complex and opaque they appear until you’ve worked with them for a bit and can sort of see the logic of their approaches. Even then the documentation takes some getting used to – the man page should just say “don’t bother, go to the website” – and even then you’ve gotta kind of fumble your way towards competence if you want to use them day to day.

Well, maybe you don’t, but I sure do. In any case once you know they exist you muddle your way to doing a lot with them. In particular, “convert” from the ImageMagick tool suite lets you upscale some of those Mario-level gifs to PNGs, like so:

$> convert mario-1-1.gif -scale 300% mario-1-1.png

We’re doing this conversion because FFMpeg (apparently?) doesn’t like to pan over gifs as an input stream but is happy to do that with PNGs, and scaling it up gets you an image size better suited for modern screens. We’re admittedly scaling up and then compressing something that eventually gets upscaled again, which looks like it should bea waste of effort. I’ve tested it, though, and on this machine at least it looks like movie upscaling comes out a lot mushier than static image upscaling and this approach is quite a bit crisper.

In any case, then you run “file” on that resulting image to see how big it is:

$> file ./mario-1-1.png
./mario-1-1.png: PNG image data, 10152 x 672, 4-bit colormap, non-interlaced

Do a bit of loose math to figure out your frame width and subtract 16/9 * 672 – that is, the aspect ration of your monitor times the height of the image – from the length – to get the number you need to work with next – in my case rounding to 1200, it’s 8952.

That’s the number of frames you’re going to tell FFMpeg to pan across, like so:

$> ffmpeg -loop 1 -framerate 5 -i mario-1-1.png -vf crop=1200:672:n:0 -frames:v 8952 -pix_fmt yuv420p mario-1-1.mp4

Now, order of operations and operation context both matters in FFMpeg usage, which adds a degree of complexity to figuring out wtf you’re doing with it, but walking through that command:

the “-loop” option is specific to the image processing part of ffpmeg, and in turn specific to some image-processing formats, so “loop 1” might or might not error out saying “unrecognized option”, depending on where you put it in the command line and which image types you’re choosing to process, which is not super helpful. In this case, it works for .png input files, and it means “go through this set of input images once”. We’ll get back to “-framerate” in a moment.

“-i” is input the png of the mario level we made earlier. The rest of this command is where the proverbial action is.

“-vf” means “create a filtergraph”, which is FFMpeg-ese for “transform the set of input images you’ve decoded in the following way”. “The following” can get pretty crazy, as you might imagine, but fortunately for us this will be reasonably simple in intent, despite the somewhat daunting syntax.

In this case, it means “crop out a sub-image from the given input image, of with 1200 and height 672, starting at horizontal offset “n” and vertical offset 0. “n” in this case is implicitly provided by the “frames” part, as we iterate over the frames from zero to the value of “-frames:v”

The “-pix_fmt yuv420p” part – “pixel format”, is what that means – I don’t really understand, beyond the fact that FFMpeg can encode videos in way more formats than browsers can easily decode, and its’ default idea of “best” doesn’t work everywhere. This incantation seems to fix that, which isn’t particularly satisfying but is definitely part of the whole fumbling-towards-competence part I mentioned.

In any case, the “-framerate 5” part is the interesting bit. That’s there because about nine thousand frames – 8952 specifically – divided by the number of seconds in a 30 minute meeting is very close to five. Five frames per second is really slow, so the resulting output video is, as predicted by our basic arithmetic, a lazy 29 minutes and 50 seconds long:

… and that’s the story of how you make a videoconference background that scrolls slowly through a Mario level over the course of half an hour.

A few notes:

  • If you leave out the framerate option and just want to see it scroll by at a default 25 frames per second, the movie is five minutes and change, which is amusingly a few seconds longer than the best speedruns of the entire game.
  • That crop=1200:672:n:0 option elides a lot of possible complexity; there’s an entire mathematical-expression interpreter under the hood of of crop and all the other FFMPeg filters, so if you want a 1080p movie panning diagonally across some of the many classic and modern works of art that are available now from any number of places, you can roll your own with relative ease.
  • The temptation to edit these to say something like “Thank you, Mario! But Peach went to another meeting.” is strong; if I get around to that, the fonts are here or maybe here.
  • I really need to get out of the house more. I guess we all do?

Update: A friend points me at FFMprovisr:

“FFmpeg is a powerful tool for manipulating audiovisual files. Unfortunately, it also has a steep learning curve, especially for users unfamiliar with a command line interface. This app helps users through the command generation process so that more people can reap the benefits of FFmpeg.”

Thank you, Sumana!

Read the whole story
smarkwell
53 days ago
reply
Share this story
Delete

People’s Homes

1 Share

“People’s Homes” is an audio project by artists Emily Fitzgerald & Molly Sherman, currently featured in ‘Hold the Phone,’ an interactive and experimental sound art and storytelling installation exploring topics vital to inner North and Northeast Portland neighborhoods.

Hear parts of “People’s Homes” on YouTube or SoundCloud.

'Hold the Phone’ is a partnership between Futel and Open Signal, funded in part by the Regional Arts & Culture Council.

Read the whole story
smarkwell
76 days ago
reply
Share this story
Delete
Next Page of Stories