TIP: Use Markdown or, <pre> for multi line code blocks / <code> for inline code.
These forums are read-only and for archival purposes only!
Please join our new forums at discourse.kohanaframework.org
Database migrations with Minion?
  • I spotted the tasks-migrations module, but am wondering how to use it (there's no Readme file). Is there a Minion task that creates the timestamped migration file that I can edit to create my DB schema within the up() method? A simple example of the process/workflow would be appreciated.

  • Install the module, use --help on the tasks that minion shows you. How to use it is in the help output.

  • ./minion migrations:new --help

    Class Task_Migrations_New contains 1 abstract method and must therefore be declared abstract or implement the remaining methods (Kohana_Minion_Task::_execute)
    
  • Maybe the module is too dated now and doesn't work with the newer changes to minion?

  • Earlier I was getting an invalid task error because the migrations module has the tasks directory within a minion directory, so I moved the tasks folder and changed the class names. Example: Minion_Task_Migrations_New was changed to Task_Migrations_New.

  • yeah, the migrations module hasn't been updated for 3.3 yet

  • @Zeelot3k I was curious if you have looked at the pull requests I have open for the migration-tasks module. Sorry, a little off topic I suppose

  • I just did and they look fine but I have to spend any free time trying to close tickets for the 3.3 release so it will have to wait until that is done :( I'll test and merge them as soon as I can

  • Is the tasks-migrations module okay for use with 3.3 now? I assume the 3.3/develop branch is the one to go for?

  • I've been using 3.3/develop for a long time now with no problems.

  • Okay, thanks! Good to know. :-)

    Who's responsible for that repository? I wonder if they could create a v3.3.0 tag, in keeping with the usual Kohana module convention... maybe I'll raise an issue...

  • I've done a bit of a revision of the userguide for this module, if anyone's interested: https://github.com/samwilson/tasks-migrations/tree/guide/guide/minion-migrations

    Pull request: https://github.com/kohana-minion/tasks-migrations/pull/42

  • Didn't have time to read through all of it but I figure it's better than no docs at all :) thanks for the help!

  • No worries. :-) Thank you for incorporating it.

  • I am working on a module inspired by Laravel 4 schema & migration module.

    https://github.com/birkir/kohana-schema

    Will be released as 3.3/master in end of june.

  • @birkir.gudjonsson

    I've skimmed over your code and don't currently see anything other then query building code so assume you haven't got to the migration parts.

    A few questions:

    1. Given you have 3 modules (A, B, C) developed and maintained by 3 independent people (X, Y, Z), and module A is dependent on things from module B and C. How is the case where constraints and/or default entries spanning multiple modules are required in table A. How is the dependence of B handled in a simple full database reset (ie. upgrade not considered).

    2. Assuming the classes in question are not just for installing. Given you have the module structure above and module A gets updated several times, and module B also gets updated some time later. How does the system know to update B? And if B has a dependency to C that's not visible in the current source tree (ie. module C is on old code), how does the system know NOT to try to update B?

    3. Assuming the system from 1) and module A has a migration A1, where it creates a basic structure and establishes a basic constraint to a column in a migration B1 from B, then B proceeds to have a migration B2 that doesn't affect A, and subsequently a migration B3 that drops the column in question. How would code in a migration A2 look to account for such a change assuming there also needs to be a A3 and so forth, and potentially a C2 from C that needs to happen in between all of this. And also how does your system maintain the history of the migrations in question (specifically what format; since going to assume it's in some database table/tables one way or another).

    4. Where is the database where the migration is suppose to live in configured in?

    5. Is it possible to inspect/check the history?

    6. Your migrations seem to be dependent on the query builder. Why? The one liners there are the same 1 liners you'd have in basic SQL or whatever other else. If the intent is to have it be database independent then how does that work when you have (a) modules that are not database independent (b) functionality that isn't database independent is added and several modules add it differently at different points in time. Or just simply: the api changes. Essentially since it seems like the builder is hotwired into the migrations what happens to old migrations when the api changes? Wouldn't that (a) break the migrations or (b) make new clean copies generated with the system be potentially incompatible with old copies generated with the system.

  • Been working on something similar and others too. Check this out: https://github.com/victorstanciu/dbv

    It's not Kohana specific, but it's a good library

  • @feketegy Nice work, I'll give it a try

  • @faketegy

    Look very nice and clean, my compliments on that.

    Some questions if you don't mind, what happens when you want to use that as part of an application you want to give to multiple people that might install it at different times and upgrade it at different times? and not just day by day thing where you're incrementing in very tiny baby steps and everyone's aware of them. Basically at some point or another you're going to have a ton of migrations for different table that will have to happen on one install. By the documentation I believe you're migrations are per table or some such and manual, so if you have a shit ton of them you're going to have a lot of conflicts between which runs in which order since obviously the migrations always work on the assumption of a particular state of other tables so you can't just try to populate a table with dependencies from another if you pushed the other table so far ahead that it doesn't even exist anymore or the state that's required of it doesn't, you can't invalidate the migration either since what's happening now is you might have another migration along the line that was responsible with moving records around and similar stuff happening (very common if the migration is a post launch/production one). Somehow I'm getting the idea you're restricting everything to one giant pile at the application level and have no support for anything that is say from a third party or the library level (ie. framework level) and also pushing any resolution to manually doing it by hand. Don't get the wrong idea, not trying to say that's an invalid way of going about it or anything, it is very pretty =P just curious of how you deal with it.

  • @Akki

    It's not my code, but to answer some of your "questions". The beauty of the whole thing that if you version control it along with the migration updates, it will work pretty nicely.

    One thing to make sure, is to never roll back, instead create a new update that's doing the "rollback".

    Another thing, is to always increment every update, don't skip for example doing update 1 and skip directly to 3. I've had a couple of nightmares doing this.

    As far as managing the individual updates and tracking changes, version controlling it, is the logical way.

  • @feketegy

    Another thing, is to always increment every update, don't skip for example doing update 1 and skip directly to 3. I've had a couple of nightmares doing this.

    So... I'll take that as a No to parallel migrations. =P

    If you don't understand what I mean here's tldr version of what I explained in my earlier post:

    feketegy creates a module A Akki creates module B (seperate independent repos; neither main application) feketegy creates a migration 1 Akki creates a migration 2 that relies on A feketegy creates migration 2 because he doesn't know of Akki's migration. ...everything breaks

  • @Akki

    That's a tricky one. Personally I've never had issues with it, because from a project management perspective every dev works on separate features in isolation as much as possible. If one dev needs to rely on a feature, hence he needs the db schema for that feature + the migrations which is already pushed in.

    And if the other dev breaks it in his new commit, than that is handled just as any other conflict.

    Usually at the beginning of the projects I see a lots of db migrations, where each individual table is created and the whole db takes shape. Then after development continues it's less and less new additions to the schema, it's more like alterations on the existing ones.

    There's a possibility for what you've said, but I never did saw it in practice. It just depends on how you manage it I guess. The migrations are just a tool to keep track of individual updates. Nothing more.

  • @feketegy

    Then after development continues it's less and less new additions to the schema, it's more like alterations on the existing ones.

    You're fortunate and lets face it "I'm never going to have to touch this [code/schema/something] ever again" is wishful thinking. =P

    There's a possibility for what you've said, but I never did saw it in practice. It just depends on how you manage it I guess. The migrations are just a tool to keep track of individual updates. Nothing more.

    I've had projects where I see it all the time and others where I see it seldom. Typically if you are not going to take out / put in a major feature of the application you won't see it. Like git it's not that everyone needs all the features (you can get away with never merging or branching in a project) but that you have the option when you inevitably will need them... or that it doesn't go to hell when you encounter "the problem".

    Also obviously it's not how you manage it, "you" can't manage it, that's the whole point. I still think you don't understand my question, the issue here is if updates to the history requires locking, synchronization and notification of all participating parties. Imagine you have any system you share between your projects, some projects need alterations to the tables of the system, some don't and at one point or another that system itself will need updating, in that scenario a singular 1.0 -> 2.0 -> 3.0 -> 4.0 either fails (as is your case since you said version skipping is now permitted) or you end up with nonsense like the migration for the system in question going use some form of brute force mitigation like 1.0 -> 1000.0 and all your other projects jumping to versions like 1001.0, 1002.0 and so forth.

  • If you share components between projects then later on those system needs to change to cope the projects' specific needs, then in my book that component is not shared anymore, hence it is handled separately.

    I don't consider db migrations any different from code versioning, it's just SQL code in a bunch of files. This may be brute and primitive, but it works, at least for me and my team.

  • @feketegy

    By shared I mean the codebase is separate and besides one modification there's no difference between them. Why maintain duzens of copies of the same code per project, when you can just maintain the small differences between them per project and just pull (and perform updates) on a single central repo. =P

  • Migrations (I've found) are often more complicated than just executing SQL. Sometimes, there're masses of files to be moved around, renamed, or maybe be scanned for metadata. I use the above module for that stuff too, and the down() method reverses whatever changes were made. Complicated, sometimes, but it means that in any change request I have a solidly tested and implementable backout plan. I used to go by the idea that any roll-back would actually be a new migration, but that makes it seem like it'll only be written if required, and that's not enough.

    Migrations often have to store data (that's being deleted or moved or whatever) so that it'll be usable by the opposite half of the migration.

    And really, I've found that the down() stuff is rarely used in production, but very often in staging.

    (Did anyone get a chance to proofread the docs I wrote above, by the way? I'm sure there's more to be said, with better grammaticalness.)

  • @samwilson

    I used to go by the idea that any roll-back would actually be a new migration, but that makes it seem like it'll only be written if required, and that's not enough.

    It is a new migration since you're still moving forward. History only moves forward, even if it regresses. The so called "down" pattern is essentially nothing more then a "history rewrite" which has the unfortunate side effect (as with git history manipulation) of potentially producing a giant pile of poop; I mean for one if you do up up down new up based on there being a down, then another version that did up up and new up is incompatible. Like with feketegy's system it's all about "you" being a history recording machine and "you" not messing it up, which means if there's more then one application using the migrations the chance for errors is very high.

    I don't believe a migration system needs anything more then a full cleanup/uninstall system. Patching past mistakes needs to still move forward, all you need to do is make sure a patch doesn't apply twice or doesn't apply after a certain point in history where it's not valid anymore.

  • It is a new migration since you're still moving forward.

    Yep, down migrations are an inherently broken idea. Don't do them.

    Your database isn't code.

  • I guess a simple upgrade Task is really all that's required. It can do all necessary checking and creating and whatnot. And get bigger and bigger over time, but that's okay.

  • Or you could... just have your migration system not have those problems? =P

  • Yep, down migrations are an inherently broken idea. Don't do them.

    Then, what should be used? When are "down" migrations useful?

  • when you are dealing with code (aka rollbacks)

  • Then, what should be used? When are "down" migrations useful?

    They are basically never useful :)

    Migrations are an inherently destructive action (adding/removing columns and tables, etc), so you can't possibly "reverse" one of those and have it make sense.

  • got it!

    thanks.

  • Personally I even keep a special settings that specifically locks any destructive migration operations out in a production environment; just in case a finger slips on the wrong keys (out of habbit in development).

    when you are dealing with code (aka rollbacks)

    Even in development setting where you have your own instance the difference between a system that has a rollback for every operation and a system that uninstalls, installs and populates with data is virtually nonexistent; except of course the rollback version is a ton more work and doesn't even work in certain circumstances (because you can't just delete data then move back forward and expect that data to magically come back).

    Also the whole rollback database before rolling back code is only intuitive for "you" like with the other problems. Someone else will just merge the rolled back code and be non the wiser of all the fidly things they have to do beforehand, since obviously your code is now at a previous version that doesn't even have the migration with the required rollback code.

  • This discussion has been very illuminating.

  • Ok, so all in all. Share your code of DB migrations :)

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

In this Discussion