Speaker notes: - Welcome everyone! Thank you for having me at Elixir Vienna. - Tonight I want to share a hands-on, practical journey deep into the Phoenix Framework test infrastructure. - If you've contributed to Phoenix or run large test suites in your own production apps, you know the pain of slow, opaque CI. - We're going to see how we took Phoenix integration tests from 10 minutes down to ~3 minutes, and the hard-won lessons along the way.

Speaker notes: - A quick 30 seconds on where I'm coming from: - For the past few years, I built a SaaS product entirely on Phoenix LiveView. - Recently, I launched Praia Labs, where I consult and help other engineering teams build, optimize, and scale production systems with Phoenix and LiveView. - Throughout this journey, I've contributed upstream to Phoenix, LiveView, Hex, and other Elixir core tooling. - I believe deeply in giving back and caring for our dependencies—when upstream tools are fast, transparent, and pleasant to work with, the entire ecosystem wins.

Speaker notes: - A mandatory slide for any CI talk! - 10 minutes is that awkward duration: too long to stare at the screen, too short to start working on a completely different deep feature. - You lose context, check social media, grab coffee, and lose flow.

Speaker notes: - Important clarification right at the start! - When people see "Phoenix testing", they usually assume it's about testing an app built with Phoenix (like testing controllers or LiveViews). - This is about testing the framework itself upstream in the core Phoenix repository. - Every release of Phoenix must guarantee that `mix phx.new` and all generators produce 100% valid, warning-free, perfectly formatted code that passes all tests across multiple Elixir, OTP, and database combinations. - BUT: the techniques we'll cover—ExUnit scheduling, CI formatting, tmpfs, container caching—are universal and apply to every team in this room.

Speaker notes: - Important to establish this boundary up front. - People often think installer tests are integration tests. - But installer/test just checks if templates render without running them. - /integration_test actually generates the whole application, boots live databases (Postgres, MySQL, MSSQL, SQLite), drops/creates databases, runs migrations, and invokes `mix test` inside the generated project.

Speaker notes: - As a long-time contributor, this was deeply personal. Every small PR meant waiting 10 minutes. - The buffering detail is subtle but brutal: ExUnit prints single dots as tests pass. But GitHub Actions buffers output. So you don't even get dots! - You just see a spinning circle with no lines emitted for nearly 10 minutes.

Speaker notes: - The classic observer effect in performance debugging! - You try to inspect what's slow using standard tooling (`--slowest`), but doing so changes the execution semantics from concurrent to serial. - So the very act of profiling made the test run vastly slower.

Speaker notes: - Now we can measure without destroying concurrency! - PR 6825 broke down the shell script into distinct CI steps, so we could see how long each phase took. - PR 6817 created SummaryFormatter, turning GitHub Step Summary into a rich observability dashboard. - Now every PR showed exactly which test modules took the most time.

Speaker notes: - Look at this Mermaid chart—this was rendered directly into GITHUB_STEP_SUMMARY! - You can immediately see the problem: 7 bars, all starting at 0, but UmbrellaAppWithDefaultsTest drags out to 8 minutes 14 seconds. - After 6 minutes, almost everything else is done.

Speaker notes: - With 4 vCPUs, BEAM has 4 schedulers, so ExUnit provides 8 parallel worker slots by default. - But Phoenix only had 7 monolithic test files! Even at T=0, we couldn't saturate the runner. - And once the quick files finished, one worker grinded away past 8 minutes while everything else sat completely idle. - That idle tail is pure wasted wall-clock time.

Speaker notes: - This is the single most important technical takeaway for anyone writing Elixir tests. - When you write `use ExUnit.Case, async: true`, you might think all `test "..."` blocks run concurrently. - No! ExUnit spawns one worker process per module. All tests inside that module run one after another in serial order. - Parallelism only happens across different test files/modules.

Speaker notes: - This is a real quote from August 31st during early architecture discussions with Steffen. - With 50+ tests, could we run more than 8 concurrently? - My immediate, rational response: "I'm not signing up for that right now :) (it takes a lot of time iterating on this, AI or not)". - It felt like too big of an undertaking with too many moving pieces. - But as engineers... the temptation to fix the inefficiency never really leaves your mind. - Which brings us directly to...

Speaker notes: - Theory: "I'll just split those 7 files into smaller files, easy afternoon project!" - Reality: Welcome to AST parsing, test matching, Elixir formatter limits, and database concurrency collisions!

Speaker notes: - This was PR #6826. - Instead of grouping by database adapter and running 10 features serially, we split each feature permutation into its own module. - Now, when a core finishes a test file, ExUnit immediately feeds it the next file from the queue of 33 modules. - Zero worker starvation.

Speaker notes: - With 53 tests in a foundational framework, manual review isn't enough. - We built an Elixir script using `Code.string_to_quoted!/1` to parse the AST of upstream/main vs HEAD. - It normalized the generated app names (replacing "pg_auth_live" with "__APP__") and verified that the test bodies were byte-for-byte identical. - Fun anecdote: The AI assistant actually made subtle edits like expanding contractions ("it'd" -> "it would") in test names! So even with an AST matcher, there was still a human element of scrutiny and hope!

Speaker notes: - If you output 33 rows in Mermaid, the chart is gigantic and unreadable on GitHub. - So we used a classic greedy interval scheduling algorithm: when a test finishes, place the next test that started after it into the same lane. - This compresses the suite into 8 tidy lanes, matching ExUnit's 8 concurrent worker slots (4 schedulers × 2)!

Speaker notes: - Look at the difference compared to the first chart! - All 8 lanes are constantly working. When one module finishes, the next one immediately fills the slot. - Total wall time dropped significantly because we eliminated the idle worker tail.

Speaker notes: - Big infrastructure shift in PR #6831. - Why run inside a nested Docker container when GitHub provides a powerful host runner? - Running on the host unlocked actions/cache (saving ~50s on dependencies) and let us mount a 4GB tmpfs RAM disk for all temporary generated apps.

Speaker notes: - This is a critical comparison. - `mix test --test-partition` is a wonderful built-in feature in Elixir. Everyone should use it for standard unit tests! - But when your tests have heavy external service dependencies, generic hashing forces every runner to start every database. - Explicit database sharding meant sqlite3 starts instantly with zero Docker overhead, and each DB shard only runs the container it actually needs.

Speaker notes: - With 8 shards, the GitHub Step Summary UI got messy. - So we built an aggregation script: each job drops a JSON artifact, and a 10-second final step joins them all into a single, clean overview table. - You can instantly see that PostgreSQL is the critical path at 3m19s.

Speaker notes: - Here are the actual numbers from GitHub Actions! - Total run started at 16:57:42 and finished at 17:01:17. - Total wall-clock time: 3 minutes 35 seconds! - PostgreSQL is the longest shard at 3m 19s, meaning that is our theoretical limit without splitting PostgreSQL further.

Speaker notes: - Let's be honest about AI tools. Everyone in the room is either using them or curious about them. - AI was incredible for generating the AST comparison script and interval scheduling logic in minutes. - But it introduced subtle bugs: for example, naming an app "app_with_postgres_adapter_auth_live" made the generated router file exceed Elixir's default 98-character line length, causing `mix format` to fail! - You must remain vigilant. Trust, but verify with automated tests.

Speaker notes: - Comic 2928 from May 2024. - Alt text: "The company tried to document how often employees were celebrating Software Testing Day, but their recordkeeping system kept mysteriously crashing." - Edge cases are what we live for in testing.

Speaker notes: - 5 practical rules you can take back to your own projects tomorrow morning. - Especially #2 and #4: check your longest test files, split them up, and use tmpfs in CI.

Speaker notes: - Thank you everyone for your time! - A huge, heartfelt thank you to Steffen Deusch (@SteffenDE). He has been doing fantastic work maintaining Phoenix and spent hours exploring partition ideas with me and reviewing these PRs. - Also thanks to Chris McCord, José Valim, and the whole Phoenix Core team for building and stewarding such an incredible framework. - Happy to take any questions!

Speaker notes: - Aaron Renner's 2021 blog post "Improving Testing & Continuous Integration in Phoenix" laid the groundwork. - Earthly had a great promise: run CI builds locally with identical containers. - But by 2024, it became a point of friction for contributors when things broke locally. - When Earthly was removed in PR #5817, what was left in CI was a monolithic `test.sh` script running inside a nested Alpine container.

Speaker notes: - Don't optimize CI at the expense of local developers! - On Apple Silicon, MSSQL crashes or crawls under Rosetta emulation. We detect ARM64 and cleanly skip MSSQL locally. - Backing containers stay warm between test runs, so iterating locally is instantaneous.

Speaker notes: - Two nice hygiene PRs that cleaned up technical debt. - The 1500ms sleep tax: 8 instances of sleeping 1.5 seconds. By backdating files on disk, we eliminated all sleeps entirely. - And the directory leak: dating back to when phoenix.new was extracted in 2014/2015. Clean teardown enabled the next big optimization: tmpfs.

Speaker notes: - Annex slide for distribution and reference. - All PRs are open or merged upstream at github.com/phoenixframework/phoenix.