Saurabh Daware 🌻

Posted on Sep 2, 2019 • Edited on Nov 2, 2019

package-lock.json vs package.json But what the hell is package-lock.json?

#node #javascript #npm

So yeah I am going to write about probably the most ignored file from our directories package-lock.json !!

package-lock.json is an extremely important file that is there to save you from a lot of boom boom bam bam 🔥 in your repositories.

So before we get into package-lock.json let's talk about semantic versioning and package.json.

1. Semantic Versioning

Semantic versioning or SemVer is the ideal way of versioning packages. They are usually written like 1.4.5 (major.minor.patch)

1a. Bug fix/patch version

Includes bug fixes/documentation spelling mistakes etc.

1b. Minor version

Includes additions of functions or API which does not break anything from the older versions So anything that runs on v1.1.0 should work on v1.9.0 as well.

1c. Major version

Includes version which breaks stuff. It can include removing APIs or changing names of functions so anything that works on v1.0.0 may not necessarily work on v2.0.0

2. Package.json

package.json is a file that contains information about your project (name, version, etc) and it lists the packages that your project is dependent on.

So as you can see in the picture above after every dependency listed under package.json there's a number something like ^2.20.0 which is the version of that package but before the version, there is ^. So ^ this little guy can be a total destroyer for your project.

^ sign before the version tells npm that if someone clones the project and runs npm install in the directory then install the latest minor version of the package in his node_modules.

So lets say I am having express with ^2.20.0 in package.json and then express team releases version 2.24.0 and now when someone clone my repo and runs npm install in that directory they will get the version 2.24.0 (You can also put ~ instead of ^ it will update to latest patch version)

However, this can be a huge issue if package developers break any of the functions on the minor version as it can make your application break down.

So npm later released a new file called package-lock.json to avoid such scenarios

3. package-lock.json

package-lock.json will simply avoid this general behavior of installing updated minor version so when someone clones your repo and run npm install in their machine. NPM will look into package-lock.json and install exact versions of the package as the owner has installed so it will ignore the ^ and ~ from package.json.

Also, it contains some other meta information which saves time of fetching that data from npm while you do npm install.

You can refer npm blog for some more information on package-lock.json.

Thank You for reading this!

I hope this was useful 🎉 :)

EDIT: So while reading the comments I thought I should also explain how package-lock.json changes so here's one of the replies that I wrote that I think everyone should go through

Saurabh Daware 🌻 • Sep 2 '19

So I created a project named 'project' and did npm install --save vue-extra@1.0.0 and cloned it three times so there's 'projectclone1', 'projectclone2' and 'projectclone3'

projectclone1

In projectclone1 I have same package.json and package-lock.json as the original project (which means I did not change anything manually) and I run npm install so it installed the same version as original that is v1.0.0 of vue-extra

projectclone2

In projectclone2 also I had the same package.json and package-lock.json but here instead of doing npm install I did npm install --save vue-extra which updated the package changing the package.json and package-lock.json so it installed the latest version that is v1.1.4 of vue-extra

projectclone3

In projectclone3 I opened package.json and manually changed vue-extra:"^1.0.0" to "^1.1.4" and did npm install, Here since I updated package.json npm considered package.json as a matter of truth and installed v1.1.4 of vue-extra and it also updated package-lock.json to v1.1.4

So if your package.json is somehow changed or updated and the version in package.json does not match with the version in package-lock.json then it will install the version from package.json and will update the package-lock.json accordingly.

I hope this clears up everything

Thanks for reading and asking this question.

EDIT2 : Quoting from the comment of Kat Marchán
(https://dev.to/zkat/comment/epbj) She is the one who wrote npm ci and added package-lock.json to NPM

Kat Marchán • Sep 3 '19

Hi! I wrote npm ci and I'm also the one who added package-lock.json to NPM back in the day.

The story about package.json vs package-lock.json is tricky: npm install does not ignore package.json versions, nor does it ignore the package-lock.json. What it does is verify that the package.json and package-lock.json correspond to each other. That is, if the semver versions described in package.json fit with the locked versions in package-lock.json, npm install will use the latter completely, just like npm ci would.

Now, ff you change package.json such that the versions in package-lock.json are no longer valid, your npm install will be treated as if you'd done npm install some-pkg@x.y.z, where x.y.z is the new version in the package.json for some-package.

This was done intentionally because, after early feedback in npm@5, we realized that one of the ways people edited their dependencies was by editing package.json directly, and it became a bit of a usability nightmare to treat package-lock.json as canonical in those cases. It was a trade-off between two competing worlds, and the current behavior won out.

This is why npm ci was born: because the behavior for npm install was actually what people wanted, in practice (when they actually ran into the behavior), and npm ci had a nice ring to it anyway (it was eventually backronymed to clean-install for this reason).

Hope this helps! Nice article! 👍🏼

</div>

Top comments (24)

Kat Marchán • Sep 3 '19

Hi! I wrote npm ci and I'm also the one who added package-lock.json to NPM back in the day.

Hope this helps! Nice article! 👍🏼

Saurabh Daware 🌻 • Sep 3 '19

Thank you so much for this comment and yes it explained everything about the npm install behavior.

also thank you for reading this it's pretty cool to see the article reached the person who added package-lock 😭😭😭🌻🌻

robby • Jan 10 '20 • Edited

Hello Kat 👋

I found this article and this comment after so much searching, and both have been a great help to my understanding of npm. Thanks everyone!

There are still somethings I am not clear on.

1) Does npm install with a package-lock present, with semver satisfied between it (the lockfile; having exact versions) and the package.json (with semver), cause the package-lock file to be updated? (maybe there's a new satisfiable version out there in the registry) -- OR does it just go about installing the modules since package.json and package-lock are in sync?

If it does update even when satisfied, does npm install some-new-package also cause this update?

2) Is it currently acceptable to manually update package.json?

3) You mentioned multiple npm install behaviors, so which are you referring to here:

the behavior for npm install was actually what people wanted, in practice (when they actually ran into the behavior)

Thanks again! Any help is greatly appreciated!

Cheers 🤙
Robby

DeLaat • Sep 2 '19 • Edited

I'm not sure this is completely correct any longer with the latest version of NPM?

From my understanding, npm install command will still use the main packages file, and updates the lock file. This is why people constantly see the lock file being updated in commits.

To use the exact versions as specified in lock file, you can use npm ci. This is intended for builds/deploys so you never get newer versions than what someone developed on.

I may be misunderstanding though...

Saurabh Daware 🌻 • Sep 2 '19

So I created a project named 'project' and did npm install --save vue-extra@1.0.0 and cloned it three times so there's 'projectclone1', 'projectclone2' and 'projectclone3'

projectclone1

projectclone2

projectclone3

I hope this clears up everything

Thanks for reading and asking this question.

DeLaat • Sep 2 '19

Interesting, thanks for taking the time to run those tests!

Out of curiosity, what version of NPM are you using? I think some of the confusion is behavior changed at some point. So, depending on what version various team members are on, they see different actions.

See this S.O. post for an example of the confusion - stackoverflow.com/questions/450220...

Saurabh Daware 🌻 • Sep 2 '19

I am using v6.11.2 and yes you are right the behavior had some issues and some changes during v5.x.x but now I guess almost all of them are fixed so v6 has been pretty stable about the behavior of package-lock.json

In the same stackoverflow answer I found this link of the issue github.com/npm/npm/issues/17979#is... which I found pretty useful.

Andrew Mackrodt • Sep 2 '19

This is how it works, author is incorrect.

Saurabh Daware 🌻 • Sep 2 '19

Hi, You should checkout this reply : dev.to/saurabhdaware/comment/eoo4
and this comment from github: github.com/npm/npm/issues/17979#is...

So yes

if you change package.json, package-lock.json will be updated
But if you do not change anything manually then it will ignore the ^ from package.json and will install the version that is mentioned in package-lock.json.

Thank you for reading and do correct me if I am wrong.

Andrew Mackrodt • Sep 2 '19 • Edited

Hi Saurabh, my post was made on mobile so please excuse it's lack of detail. I've experienced package-lock.json updating itself in both development and CI environments and subsequently builds have failed when sub-dependencies have introduced bugs or breaking changes despite following semantic versioning. npm ci faithfully installs the correct package versions in the CI environment.

As to why this happens, I'm not sure. I agree that what you've said should work yet in practice I've seen it to not always be the case.

The npm team seem to acknowledge this, here's a quote from their blog:

npm ci promises the most benefit to large teams. Giving developers the ability to "sign off" on a package lock promotes more efficient collaboration across large teams, and the ability to install exactly what is in a lockfile has the potential to save tens if not hundreds of developer hours a month, freeing teams up to spend more time building and shipping amazing things.

blog.npmjs.org/post/171556855892/i...

It seems hundreds of developers on SO are also confused about npm's lock behaviour: stackoverflow.com/questions/450220....

Saurabh Daware 🌻 • Sep 3 '19

Yeah even ive ran into problems where 100s of lines were updated in my package-lock.json so i think it is because a lot combinations are possible like i mentioned above plus if you've seen dependabot commits, they update package-lock to bumb versions but along with that they also change the integrity hash so it doesn't end up creating conflicts.

also a lot of time we pull from other branches so if any of them updated your dependent package they may end up updating the tree in package-lock

so yeah a lot of permutations and combinations to think about :(

Taimoor khan • Oct 14 '20

Hi Saurabh Thanks for the super easy explanation of Package-lock.json.
Right now I am learning React and every time I have to add new dependency in my app I use npm install somePackageName and the get the error
npm ERR! Cannot read property 'match' of undefined
so every time I delete package-lock.json file and again run the same command and this time with success result. This is what brought me here

Saurabh Daware 🌻 • Oct 14 '20

That's a weird error. Maybe you should try having --save-dev or --save flag on npm install. Not sure about what the default npm install behavior is but normall --save or --save-dev will add it to package.json dependency object and it will be installed locally.

Taimoor khan • Oct 14 '20

tried that too but that didn't solved the problem...! I had to delete package-lock.json and then execute the command again...

Peter DeMarco • Sep 2 '19

In order for npm to behave as described in the article (honoring versions from package-lock instead of just the latest that will satisfy package.json) you have to run npm ci instead of npm install FYI. docs.npmjs.com/cli/ci.html

Saurabh Daware 🌻 • Sep 2 '19

npm ci will install from package-lock.json ONLY. So if you manually change the version from package.json it will throw an error.
npm install on the other hand will install from package-lock.json as long as package-lock.json and package.json are in sync. If in any case, package.json has changed then it will install version from package.json and will update the package-lock.json accordingly.

Thanks for reading and pointing out npm ci. Do correct me if I am wrong.

Melroy's Blog • Sep 2 '19 • Edited

Please note that devs forget to add this important file to the version control system (read: git). So please devs, add this file. And update when needed (especially for security issues).

Bottom line: do NOT gitignore the package-lock.json file.

Saurabh Daware 🌻 • Sep 3 '19

yeah exactly, thanks for pointing out :)

Wernich ️ • Sep 2 '19 • Edited

this is so far the best explanation i've read of what this file does. i'm assuming the reason why this file changes so much in our repo is because people are constantly upgrading npm packages? we've had many merge conflicts around that file, so it eventually got added to gitignore.

i didn't hear the whole conversation, but it sounded like one of our test deploys failed miserably because of npm package changes somewhere. could probably have been avoided then.

Saurabh Daware 🌻 • Sep 2 '19 • Edited

I think my this comment will explain a lot of extra things dev.to/saurabhdaware/comment/eoo4

So basically there are few cases when package-lock.json changes

You npm install --save <package> so it updates the package to absolute latest version changing the version inside package.json and package-lock.json
When package.json changes: So when you (or let's say dependabot or your other teammate) changes the version of the package in package.json npm install will look into package.json for the version and will update the package-lock.json with respect to package.json

Also gitignoring package-lock is kind of risky unless you have any other workaround it

And thank you for reading! I am super happy to see that you found it useful :D