Have you ever seen Twitch Plays Pokemon (TPP) and wondered how it actually works? How does typing comments in a twitch stream result in the player moving in the original Pokemon Games, how is the screen directly streaming to twitch 24/7? In this post we aim to provide a few answers to these questions and more.
If you have never seen TPP check it out live on twitch: TwitchPlaysPokemon - Twitch
The original implementation of Twitch Plays Pokemon was developed in and was closed source (probably due to security concerns of having a twitch chat control your computer..).
It was mainly implemented in python but a overlay was also developed with Javascript and the React Framework 1.
So it consisted of a few main parts:
In the original implementation it used the glob al keyboard event in the python win32 API like so:
win32api.keybd_event(keymap[button], 0, 0, 0)
The downside of this is the window needs to be in focus to get the event, otherwise if the focus is changed it will start sending the input to other windows running in the OS, not ideal from a security standpoint.
This was later changed to directly interface with a Lua script running in the emulator that connects via HTTP, so the button presses go directly to the emulator and this also has a few advantages such as frame perfect timing.
According to the AMA (Ask Me Anything) thread on reddit with the creator it requires two servers, one to compress the video and the other to do everything else such as play the game and display the overlay. It would be interesting to find out what the video compression server is doing that couldn’t be done with standard OBS on the same server.
Twitch plays pokemon has supported multiple Nintendo console emulators such as:
Sam Agnew who works at Twilio gave a keynote at the 12th Python Brasil conference in 2017 discussing how to create a similar experience to TPP using Python!
The video starts of with introducing the speaker in Portuguese as the conference was hosted in Brazil, however if you don’t understand Portuguese then don’t worry the main content of the presentation is in English.
This contains:
This is the lua source code that he wrote live during the keynote, you can run it in the windows version of VisualBoyAdvance (for Mac/Linux you need to use WINE).
function file_exists(name)
local f = io.open(name, 'r')
if f == nil then
io.close(f)
return true
end
return false
end
function read_file(name)
if file_exists(name) then
local input = io.open(name, "-")
io.input(input)
content = io.read()
io.close(input)
return content
end
return nil
end
function press_button(button)
local input_table = {}
input_table[button] = true
joypad.set(1, input_table)
end
while true do
if file_exists('button.txt') then
button = read_file('button.txt')
os.remove('button.txt')
emu.message('Pressing button:')
for i=0, 5 do
press_button(button)
emu.frameadvance()
end
end
emu.frameadvance()
end
With this script you can manually create a button.txt file with one button press per line or even better you can write a script that generates button.txt based on some external event such as a Twitch chat.
The source code for a similar project covered in the Keynote is available on Github: sagnew/Twilio-IP-Messaging-Plays-Pokemon: A clone of the 2013 phenomenon “Twitch Plays Pokemon” using Twilio’s new IP Messaging API.
Back in 2014 TwitchPlaysPokemon was completing game after game and soon the next in line would be a 3DS Pokemon game, but without good 3DS emulation back then how exactly would TPP continue?
The answer is custom hardware created by a hacker called dekuNukem who modified a retail 3DS console, added a capture card and streamed the 3DS game directly to twitch for the next year! 2
This hardware was named 3xtDS and allows full control over all the buttons and even the touch screen over a standard USB interface 3.
This means that any code running on the host PC can control the 3DS and see the result via the capture card, so no custom code needs to be written for the 3DS CPU itself, so the 3DS does not need to be jailbroken.
You can see the hardware in action in the video below:
You can also see a close up of the device and hardware modifications in the Album on Imgur: 3xtDS - Imgur
In the video you can see that text commands can be sent directly to the device to control the game using the USB interface, very impressive!
The source code is also available on Github if anyone has the necessary experience to create their own: dekuNukem/3xtDS: 3DS Streaming Console with External Control Interface.
When I first heard about Twitch Plays Pokemon and other similar projects I had presumed it was running a script which streams the emulator screen directly to twitch. However the more I look into it, it seems they all are running standard PCs with a GUI such as OBS for streaming.
This was a little disappointing as it would be nice to have some sort of container (such as a docker image) that could be deployed to a base server (or “the cloud”) with minimal overhead.
So it begs the question, is it possible to create a completely headless (No GUI) version of TPP so its easy to host a very lightweigt version on fairly cheap hosting solutions?
So Twitch receives its streams in the Real-Time Messaging Protocol (RTMP) so if we can send data directly using this protocol from an emulator we could in theory cut out the middle man and not require a GUI desktop environment running on the server.
One solution could be using the ffmpeg command line interface, but the question is how exactly do we combine that with the output of our emulator…
To run a test stream from ffmpeg to twitch you can use the following command after adding your own stream key to it:
ffmpeg -re \
-f lavfi -i testsrc2=size=960x540 \
-f lavfi -i aevalsrc="sin(0*2*PI*t)" \
-vcodec libx264 \
-r 30 -g 30 \
-preset fast -vb 3000k -pix_fmt rgb24 \
-pix_fmt yuv420p \
-f flv \
rtmp://live-fra.twitch.tv/app/STREAMKEY
howitworks
howitworks
gameboy
gameboy
gameboy
gameboy
gameboy
gameboy
gameboy