Slides used during Meet Magento Netherlands 2023 to discuss the speed at which Magento integration tests are being run. Save the Dutch!

1. #MM23NL
FASTER MAGENTO
INTEGRATION TESTS
if you wait too long,
you are going to do something else

2. #MM23NL
Jisse Reitsma
- Founder of Yireo
- Training Magento developers
- Maintaining open source extensions
- Board member of Mage-OS Nederland
- Organizer of MageUnconf NL
- Dad of currently 1 but soon 2

3. #MM23NL
ASSUMPTIONS
- You are familiar with Magento backend development
- You know what integration tests are
- You have tried running (any) tests at least once

4. #MM23NL
#MM23NL
QUESTION: WHO
IN THIS ROOM
WRITES
INTEGRATION
TESTS?

5. #MM23NL
42

6. #MM23NL
#MM23NL
QUESTION: HOW
LONG DO YOU
USUALLY WAIT FOR
INTEGRATION
TESTS TO BE
FINISHED?

7. #MM23NL
1m 20s

8. #MM23NL
WHY FASTER INTEGRATION TESTS
- Faster feedback loop

9. #MM23NL
WHY FASTER INTEGRATION TESTS
- Faster feedback loop
- Less barriere to get started with testing

10. #MM23NL
WHY FASTER INTEGRATION TESTS
- Faster feedback loop
- Less barriere to get started with testing
- Less energy consumption

11. #MM23NL
LESS POWER NEEDED

12. #MM23NL
REDUCE CLIMATE CHANGE

13. #MM23NL
PREVENTING THE RISE OF THE SEA LEVEL

14. #MM23NL
SAVE THE DUTCH

15. #MM23NL
SAVE THE DUTCH!
FASTER TESTS!

16. #MM23NL

17. #MM23NL
#MM23NL
GETTING THE FILES

18. #MM23NL
GETTING MAGENTO
$ composer create-project
--repository-url=https://repo.magento.com/
magento/project-community-edition .

19. #MM23NL
GETTING MAGENTO MAGE-OS
$ composer create-project
--repository-url=https://repo.mage-os.org/
mage-os/project-community-edition .

20. #MM23NL
FASTER COMPOSER DOWNLOADS
- Use a Mage-OS repository nearby
- Create your own mirror
- mage-os/generate-mirror-repo-js
- Private Packagist
- Satis

21. #MM23NL
TUNING COMPOSER CACHE
- Folder $COMPOSER_HOME/cache
- Try to reuse the cache amongst various installations
- S3 bucket
- Docker volume
- ZIP artifact to unzip before install

22. #MM23NL
#MM23NL
SETTING UP
INTEGRATION TESTS

23. #MM23NL
SETTING UP INTEGRATION TESTS (1)
- Navigate to dev/tests/integration/

24. #MM23NL
SETTING UP INTEGRATION TESTS (2)
- Navigate to dev/tests/integration/
- Copy phpunit.xml.dist to phpunit.xml

25. #MM23NL
FILE phpunit.xml
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="http://schema.phpunit.de/9.6/phpunit.xsd" colors="true"
beStrictAboutTestsThatDoNotTestAnything="false" bootstrap="./framework/bootstrap.php" stderr="true">
<php>
….
<const name="TESTS_CLEANUP" value="enabled"/>
<const name="TESTS_MEM_USAGE_LIMIT" value="8192M"/>
…
<const name="TESTS_MAGENTO_MODE" value="developer"/>
</php>
<listeners>
<listener class="MagentoTestFrameworkEventPhpUnit"/>
<listener class="MagentoTestFrameworkErrorLogListener"/>
</listeners>
</phpunit>

26. #MM23NL
FILE phpunit.xml
- TESTS_CLEANUP=enabled at first run, disabled at subsequent runs
- TESTS_MEM_USAGE_LIMIT=4092M (PHP memory limit)
- TESTS_PARALLEL_RUN=1
- Remove Yandex listener if it is there

27. #MM23NL
SETTING UP INTEGRATION TESTS (3)
- Navigate to dev/tests/integration/
- Copy phpunit.xml.dist to phpunit.xml
- Copy etc/install-config-mysql.php.dist to etc/install-
config-mysql.php

28. #MM23NL
FILE etc/install-config-mysql.php
return [
'db-host' => 'localhost',
'db-user' => 'root',
'db-password' => '123123q',
'db-name' => 'magento_integration_tests',
'db-prefix' => '',
'backend-frontname' => 'backend',
'search-engine' => 'opensearch',
'opensearch-host' => 'localhost',
'opensearch-port' => 9200,
'admin-user' => MagentoTestFrameworkBootstrap::ADMIN_NAME,
'admin-password' => MagentoTestFrameworkBootstrap::ADMIN_PASSWORD,
'admin-email' => MagentoTestFrameworkBootstrap::ADMIN_EMAIL,
'admin-firstname' => MagentoTestFrameworkBootstrap::ADMIN_FIRSTNAME,
'admin-lastname' => MagentoTestFrameworkBootstrap::ADMIN_LASTNAME,
'amqp-host' => 'localhost',
'amqp-port' => '5672',
'amqp-user' => 'guest',
'amqp-password' => 'guest',
'consumers-wait-for-messages' => '0',
];

29. #MM23NL
FILE etc/install-config-mysql.php
- Modify MySQL database
- Modify OpenSearch / ElasticSearch
- ...

30. #MM23NL
SETTING UP INTEGRATION TESTS (4)
- Navigate to dev/tests/integration/
- Copy phpunit.xml.dist to phpunit.xml
- Copy etc/install-config-mysql.php.dist to etc/install-
config-mysql.php
- Run ../../../vendor/bin/phpunit with your tests

31. #MM23NL
SETTING UP INTEGRATION TESTS (4)
- Navigate to dev/tests/integration/
- Copy phpunit.xml.dist to phpunit.xml
- Copy etc/install-config-mysql.php.dist to etc/install-
config-mysql.php
- Run ../../../vendor/bin/phpunit with your tests
Don't use bin/magento dev:tests:run integration because it's slower

32. #MM23NL
MY COMMAND
docker-compose exec -T php-fpm bash <<EOF
cd dev/tests/integration;
php -d memory_limit=-1 ../../../vendor/bin/phpunit
-c ../quick-integration/phpunit.xml
app/code/Yireo/Example/Test/Integration/
EOF

33. #MM23NL
1m 20s

34. #MM23NL
DISCLAIMER
If TESTS_CLEANUP is disabled (so the setup is skipped), the total time
goes down to 4-8s (unoptimized) or even 1-3s (optimized). This talk is
not necessarily about that, even though all of the following tips do
apply.

35. #MM23NL
1m 20s
Let’s get this number down

36. #MM23NL
TEST RUN BENCHMARK
- PHP 8.2
- MySQL 8
- Magento default install (of 1419 modules)
- Yireo GoogleTagManager (27 tests, 127 assertions)
- Reset of entire Docker stack, tmp/, file cache, generated/
This is run on some computer with some CPU and some RAM: Exact
numbers are not important, relative comparison is important

37. #MM23NL
HOW ARE INTEGRATION TESTS RUN
- PHPUnit calls bootstrap file
- Bootstrap kickstarts Magento application
- Magento database setup is run (setup:install)
- Or a restore from existing dump is done
- Various test events are triggered (startTest, endTest, ...)
- Tests are run

38. #MM23NL
#MM23NL
REACHDIGITAL
TEST FRAMEWORK

39. #MM23NL
REACHDIGITAL TESTFRAMEWORK
- Drop-in replacement of Magento TestFramework
- Disable memory cleanup scripts
- Fix overzealous app reinitialisation
- Disable config-global.php by default
- Disabled sequence table generation
$ composer require --dev reach-digital/magento2-test-framework
See https://github.com/ho-nl/magento2-ReachDigital_TestFramework

40. #MM23NL
1m 10s
Let’s get this number down
Before: 1m 20s

41. #MM23NL
#MM23NL
MODIFYING
INSTALLER
ARGUMENTS

42. #MM23NL
YIREO INTEGRATION TESTING HELPER
- Traits for usage in tests
- Command to toggle TESTS_CLEANUP
- Command for test setup validation
- InstallConfig and DisableModules classes
$ composer require --dev yireo/magento2-integration-test-helper
See https://github.com/yireo/Yireo_IntegrationTestHelper

43. #MM23NL
RUNNING THE MAGENTO INSTALLER
$ bin/magento setup:install –help
- Database is validated (unless --skip-db-validation is used)
- Database setup scripts are run (Setup/, db_schema.xml)
- Configuration files are created (app/etc/*.php)
- Search engine is validated
- ...

44. #MM23NL
FILE etc/install-config-mysql.php
return [
'db-host' => 'localhost',
'db-user' => 'root',
'db-password' => '123123q',
'db-name' => 'magento_integration_tests',
'db-prefix' => '',
'backend-frontname' => 'backend',
'search-engine' => 'opensearch',
'opensearch-host' => 'localhost',
'opensearch-port' => 9200,
'admin-user' => MagentoTestFrameworkBootstrap::ADMIN_NAME,
'admin-password' => MagentoTestFrameworkBootstrap::ADMIN_PASSWORD,
'admin-email' => MagentoTestFrameworkBootstrap::ADMIN_EMAIL,
'admin-firstname' => MagentoTestFrameworkBootstrap::ADMIN_FIRSTNAME,
'admin-lastname' => MagentoTestFrameworkBootstrap::ADMIN_LASTNAME,
'amqp-host' => 'localhost',
'amqp-port' => '5672',
'amqp-user' => 'guest',
'amqp-password' => 'guest',
'consumers-wait-for-messages' => '0',
];

45. #MM23NL
FILE etc/install-config-mysql.php (YIREO)
<?php
use YireoIntegrationTestHelperUtilitiesInstallConfig;
return (new InstallConfig())
->addDb('mysql', 'magento2', 'magento2', 'magento2')
->get();

46. #MM23NL
#MM23NL
OPTIMIZING
THE STACK

47. #MM23NL
OPTIMIZE TEMPORARY FOLDERS
- Mount /tmp as tmpfs
- Mount dev/tests/integration/tmp/ as tmpfs
- Possibly mount vendor/ as tmpfs
- Run MySQL in tmpfs

48. #MM23NL
#MM23NL
OPTIMIZING
MYSQL

49. #MM23NL
MY OWN docker-compose.yaml
services:
mysqltmpfs:
image: mysql:8.0
ports:
- 3306
volumes:
- ./mysqld_test.cnf:/etc/mysql/mysql.conf.d/mysqld.cnf
tmpfs:
- /tmp
- /var/lib/mysql

50. #MM23NL
MY OWN mysqld_test.cnf
[mysqld]
innodb_buffer_pool_size=512M
innodb_buffer_pool_instances=1
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_lock_wait_timeout=50
innodb_doublewrite=0
innodb_flush_log_at_trx_commit=0 # or 2
innodb_flush_log_at_timeout=360
innodb_flush_method=nosync

51. #MM23NL
#MM23NL
OPTIMIZING
PHP

52. #MM23NL
OPTIMIZE PHP
- Enable opcache
- Disable xdebug
- Set realpath_cache_size=4M

53. #MM23NL
#MM23NL
ADD
REDIS

54. #MM23NL
ADD REDIS TO etc/install-config-mysql.php
--session-save-redis-host=redis
--session-save-redis-db=0
--cache-backend-redis-server=redis
--cache-backend-redis-db=0
--page-cache-redis-server=redis
--page-cache-redis-db=1

55. #MM23NL
ADD REDIS TO etc/install-config-mysql.php (YIREO)
$installConfig->addRedis('redis')

56. #MM23NL
#MM23NL
OPTIMIZING
ELASTICSEARCH

57. #MM23NL
SEARCH ENGINE VALIDATION
Could not validate a connection to Elasticsearch. Verify
that the Elasticsearch host and port are configured
correctly.

58. #MM23NL
OPTION 1: USE MYSQL LEGACY SEARCH
Use https://github.com/swissup/module-search-mysql-legacy
--search-engine=mysql

59. #MM23NL
OPTION 2: FAKE VALIDATION
HEAD request to http://localhost:9200/ should return HTTP/1.1 200 OK
Create a simple HTML page, configure webserver to listen to port 9200

60. #MM23NL
OPTION 3: HACK THE CORE
Create a composer patch for MagentoElasticsearchSetup
Validator::validate() and always return true
Note: A DI plugin does not work. We don't have a working application yet.

61. #MM23NL
OPTION 4: FACE IT
Just use OpenSearch / ElasticSearch anyway and deal with it.

62. #MM23NL
ADD ELASTICSEARCH TO etc/install-config-mysql.php
--search-engine=opensearch
--opensearch-host=localhost
--opensearch-port=9200
Or:
--search-engine=elasticsearch7
--elasticsearch-host=localhost
--elasticsearch-port=9200

63. #MM23NL
TUNE ELASTICSEARCH
- Set discovery.type=single-node
- Set data folder to tmpfs
- When OpenSearch, disable optional plugins
- Tune Xms and Xmx

64. #MM23NL
0m 45s
Let’s get this number down
Before: 1m 10s

65. #MM23NL
#MM23NL
SKIPPING
MODULES

66. #MM23NL
SKIPPING MODULES
- Adding the flag –disable-modules to the installer
- Or use Yireo Integration Helper DisableModules class
- Or use composer replace

67. #MM23NL
YIREO COMPOSER REPLACE TOOLS
Use yireo/magento2-replace-tools to manage replacements via the CLI (including bulk
replacements and validation)
$ composer require yireo/magento2-replace-tools
$ composer replace:bulk:add yireo/magento2-replace-bundled
$ composer replace:bulk:add yireo/magento2-replace-graphql
$ composer replace:build
$ rm -rf vendor/ composer.lock
$ composer install

68. #MM23NL
USING –disable-modules
--disable-modules=
Magento_InventoryInStorePickupGraphQl,Magento_InventoryInStorePickupQuo
teGraphQl,Magento_InventoryQuoteGraphQl,Magento_CatalogInventoryGraphQl
,Magento_LoginAsCustomerGraphQl,Magento_NewsletterGraphQl,Magento_Graph
QlCache,Magento_PaymentGraphQl,Magento_ReCaptchaWebapiGraphQl,Magento_R
elatedProductGraphQl,Magento_ReviewGraphQl,Magento_SalesGraphQl,Magento
_UrlRewriteGraphQl,Magento_SendFriendGraphQl,Magento_InventoryGraphQl,M
agento_CatalogCmsGraphQl,Magento_SwatchesGraphQl,Magento_TaxGraphQl,Mag
ento_ThemeGraphQl,Magento_CatalogUrlRewriteGraphQl,Magento_PaypalGraphQ
l,Magento_VaultGraphQl,Magento_WeeeGraphQl,Magento_WishlistGraphQl

69. #MM23NL
USING –disable-modules (YIREO)
$disableModules = (new DisableModules(__DIR__.'/../../../../'))
->disableAll()
->enableMagento()
->disableMagentoInventory()
->disableGraphQl()
->enableByName('Yireo_GoogleTagManager2');
return (new InstallConfig())
->setDisableModules($disableModules)
->get();

70. #MM23NL
MY OWN etc/install-config-mysql.php
use YireoIntegrationTestHelperUtilities{DisableModules, InstallConfig};
$disableModules = (new DisableModules(__DIR__.'/../../../../'))
->disableAll()
->enableMagento()
->disableMagentoInventory()
->disableGraphQl()
->enableByMagentoModuleEnv();
return (new InstallConfig())
->setDisableModules($disableModules)
->addDb('mysqltmpfs', 'magento2', 'magento2', 'magento2')
->addRedis('redis')
->addElasticSearch('opensearch', 'elasticsearch')
->get();

71. #MM23NL
0m 38s
Before: 0m 45s

72. #MM23NL
0m 38s
Let’s get this number down?

73. #MM23NL
SAVE THE DUTCH!
FASTER TESTS!

74. #MM23NL
Yireo upcoming
video courses:
- Magento 2 Testing
- Hyvä Checkout Development
expected end of 2023 / early 2024

75. #MM23NL
#MM23NL
Thanks

76. #MM23NL