Edit:
I went ahead and ran some performance tests again with optimizations with better results.
Preface
This is the first post in a series of articles I plan to write where I will be looking at a variety of server side web frameworks or using no framework in several languages.
Goals
My primary goal is to highlight a few server side frameworks and to give enough information so that you, the reader, can make a judgement as to whether or not you should do more investigation for your use case.
Here are some goals:
- Test flexibility
- Does it impede dev?
- SQL compatibility
- Can you use raw sql
- Is the ORM, if present, good for almost any type of query?
- Quick response time performance for a multi join api endpoint
- How many concurrent users can the framework support before response time is too high for good UX.
Setup
I have done everything in docker to make it easy to keep things separated. For load testing I will be using k6.
Repo: https://github.com/buphmin/symfony4-impressions
Load tester:
docker pull loadimpact/k6
As a reference for the performance portion here is how I am testing
Hardware/Hardware Config
My hardware is faster than the majority of people so your mileage may vary on the benchmarks. I am also running everything within a linux mint 19 VM. Specs are as reserved by the VM.
i7 8700k running at 4.9ghz 6 cores
8GB ram running at 3600mhz
Cores 0-1 reserved for the web server
Cores 2-3 reserved for the database
Cores 4-5 reserved for k6
Impressions
So as a disclaimer I have worked with Symfony for many years, so this is hardly a first impression for me. I do feel, however, that I can objectively qualify Symfony's strengths and weaknesses in a way that you can determine whether or not it warrants your attention. You should always judge a technology for yourself and not blindly think it is good just because (insert famous person here) said so.
What is Symfony
Symfony is a full featured full stack framework designed to give you an all in one solution for web development. Symfony attempts to be a one size fits all type of framework with built in authentication, server side rendering/templating, api support, etc.
Flexibility
In terms of flexibility Symfony is quite flexible, especially for a fully featured framework. That does not mean Symfony is perfect though. You have to live with the config files that Symfony requires and set them up in the Symfony way. Additionally by default Symfony expects you to put your files in specific locations forcing you to
Symfony allows you to override many of it's conventions but you will likely find troublesome. For instance you can override the directory structure, but then likely be confused when the docs assume you are using the defaults.
For SQL Symfony ships with Doctrine for use of both the ORM and DBAL (DataBase Abstraction Layer). Both can be used exclusively or simultaneously. The doctrine ORM is pretty good and will allow you to use most database structures. A notable weakness is composite foreign key relations as primary keys which either cannot be done or will take some creativity to work around. The doctrine mapping can take some time to setup but Symfony comes with some tools to make the process simpler if you have already structured your database. On the flip side you can use the Doctrine DBAL for raw SQL. The DBAL is just a light wrapper around PDO, PHP's native database connection library that provides some convenience methods. Symfony also has the ability to use other connection interfaces, such as to connect to MongoDB, but the Symfony docs shy away from that and I will not cover it here. Between the the ORM and DBAL you have ways to work with a SQL database however you need.
Here are some examples of raw SQL vs ORM endpoints with multi join:
/**
* @param integer $id
* @param Connection $conn
* @return JsonResponse
* @Route(path="/league_players/raw/{id}", name="league_player_raw")
*/
public function getLeaguePlayerRawAction($id, Connection $conn)
{
$sql = "
select * from
league_player lp
join league l on lp.league_id = l.id
join player p on lp.player_id = p.id
where lp.id = :id
";
$leaguePlayers = $conn->fetchAll($sql, [
'id' => $id
]);
if (count($leaguePlayers) === 0) {
throw new NotFoundHttpException('Player not found');
}
return new JsonResponse($leaguePlayers[0]);
}
/**
* @param LeaguePlayer $leaguePlayer
* @param SerializerInterface $serializer
* @return Response
* @Route(path="/league_players/{leaguePlayer}", name="league_player")
*/
public function getLeaguePlayerAction(LeaguePlayer $leaguePlayer, SerializerInterface $serializer)
{
$data = $serializer->serialize($leaguePlayer, 'json');
return new Response($data);
}
To end the flexibility section it is interesting to note that the Symfony framework is build from a motley of independent reusable components. Each of these can be used outside of the Symfony framework as a whole. For instance if you just need a router utility then you can use the Symfony routing component and manage the rest of the stack yourself.
Performance
I will be judging the two endpoints above to judge the raw SQL implementation and the ORM implementation. The ORM implementation is slightly different, but is a realistic implementation of what you might do with an ORM endpoint.
1 Concurrent User
Raw:
docker run --network host --cpuset-cpus "4-5" -v $PWD/loadTests:/src -i loadimpact/k6 run --vus 1 --duration 10s /src/getLeaguePlayerRaw.js
Result:
http_req_duration..........: avg=4.51ms min=3.51ms med=4.46ms max=5.76ms p(90)=5.09ms p(95)=5.42ms
Pretty good performance, 11.73ms duration. Obviously since this is local to local this is not exactly "real" world performance but it gives us a rough idea.
ORM:
docker run --network host --cpuset-cpus "4-5" -v $PWD/loadTests:/src -i loadimpact/k6 run --vus 1 --duration 10s /src/getLeaguePlayer.js
Result:
http_req_duration..........: avg=16.74ms min=15.06ms med=15.98ms max=21.15ms p(90)=19.43ms p(95)=20.29ms
The ORM implementation is close to double the request time in this scenario.
10 Concurrent Users
Change --vus to 10
Raw:
http_req_duration..........: avg=8.81ms min=3.24ms med=8.35ms max=19.11ms p(90)=12.5ms p(95)=17.18ms
ORM:
http_req_duration..........: avg=51.44ms min=14.81ms med=51.05ms max=82.89ms p(90)=76.4ms p(95)=80.5ms
The extra load of the ORM method is becoming evident.
50 Concurrent Users
Change --vus to 50
Raw:
http_req_duration..........: avg=34.75ms min=4.57ms med=31.57ms max=90.81ms p(90)=58.4ms p(95)=76.5ms
ORM:
http_req_duration..........: avg=201.94ms min=15ms med=216.65ms max=1.69s p(90)=334.99ms p(95)=369.24ms
Edit:
Performance is pretty good with raw SQL in my tests at 50 concurrent users. I will note that I saw a very wide variation between runs. Most runs were between 28-45ms but some shot to 400ms, which could just be the limitations of my testing environment. With the implemented optimizations performance I was able to get around 130ms response time at 200 concurrent users. This is good but still trails behind anything in node.js as we will see in future articles ;)
Here we are at the limit of what 2 cores can achieve. 500ms is too much for good UX in most circumstances. We find that the ORM adds extra overhead over the raw SQL implementation which is to be expected.
Results
My impression of Symfony is that it quite flexible and gives you a lot of ways to go about your business.
Pros
- flexibility
- good docs
- mostly stays out of your way
Cons
- some limitations with the ORM
- high concurrent user performance is not great
Recommendation:
I would say Symfony is worth taking a look at for most people who are interested in a framework. It offers good flexibility and pretty good low load performance which makes it ideal for an internal company app.
Let me know if you have any insights or questions as this will be my first fairly in depth article.
Top comments (2)
Nice to see a Symfony article on dev.to ;) I plan to write something about it soon as it is my favorite framework.
Some comments:
Regarding your performance results, have you optimized your application for production according to this?
This has tremendous impact on performance, mostly the optimization of the autoloader.
I would also recommend replacing Apache with Nginx.
And would be nice to know, how many time of your requests is spent on the DB itself. From your slowest requests with many concurrent users, the bottleneck might be the DB itself and not the framework.
About the ORM performance, I think we would need to dig deeper to understand what is slowing it down. It could be the query itself or the object hydration. But If I understand your query, is always returning only 1 result from your "league_player_table" right? Doesn't really make sense that performance difference.
Still, I have not seen your code in detail, but I am curious. I might try something if I have some time.
ORMs are a great tool, but need to be understood what they are doing under the hood.
Loading 10.000 objects into memory is not a viable option, for example ;)
Well, to be correct Symfony 4 doesn't ship with any Persistence adapter by default. Yes, Doctrine its the most common and I would say recommended but you can easily use anything you want, even Laravel´s Eloquent.
This is one of the beauties of Symfony. Start small and add more things as you need need instead of the other way around.
Keep writing ;)
Opcache is enabled to the default settings which is the biggest boost to performance. I could change that to allow more files. I did just now try out composer optimize and got around 38ms at 50 concurrent users for raw sql, huge improvement! I have tested a barebones golang implementation of this type of endpoint awhile back with far far more many concurrent connections before the database started to choke, but I will look into it more just in case. Query time is about .5ms in this scenario.
One note for the ORM is it initiates 3 queries vs 1 here for a total query time of about 1.5ms and has to hydrate 3 objects I believe. I tried to make the endpoint something that might be a common scenario rather than a 100% optimized one. Perhaps that is not entirely fair though.
Either way it seems I need to update the performance section (and my repo) with some tweaked configuration. Thanks!