Jan 18 13:17:16 <jorrit> Xordan: in your proposal. Can you still handle the situation where you want to have all materials+textures in some collection? In that case there is no real top-level container for those textures.
Jan 18 13:18:10 <jorrit> Materials are also not tied to sectors (same as mesh factories)
Jan 18 13:18:15 <jorrit> So the remark added by res2k goes for those as well.
Jan 18 13:20:04 <res2k> though you could say that if an object doesn't have a top-level container, add the object itself.
Jan 18 13:20:40 <res2k> there are also other border cases
Jan 18 13:20:42 <jorrit> Possibly but I wonder what that solves.
Jan 18 13:20:44 <jorrit> I mean
Jan 18 13:20:54 <jorrit> If I look at that proposal it really doesn't sound that much different from regions now.
Jan 18 13:21:05 <res2k> such as a mesh object in "limbo" - ie it's not in any sector at the moment, but may be added to one at some point
Jan 18 13:22:13 <res2k> jorrit: still we should try to solve the shortcomings we perceive in regions
Jan 18 13:22:18 <jorrit> yes true
Jan 18 13:22:26 <jorrit> But I'm not sure this proposal is doing that (right now).
Jan 18 13:22:27 <res2k> if it turns out that we can "fix" regions without adding something new, well, awesome[[BR]] Jan 18 13:22:39 <jorrit> I don't want to save region at all cost. It has a bad name anyway.
Jan 18 13:22:49 <jorrit> Just that when we add something new it has to solve the problems :-)
Jan 18 14:01:43 <Xordan> jorrit: Yes, I mentioned that case
Jan 18 14:02:03 <jorrit> Xordan: I guess I don't fully understand how that proposal is going to solve the problems we have.
Jan 18 14:03:52 <Xordan> Well basically atm there's no guarantees on anything when we want to unload. More than that, it's very much a 'users problem' as to what gets unloaded and what doesn't (most things don't).
Jan 18 14:04:18 <jorrit> Ok, and how does your proposal solve that?
Jan 18 14:04:50 <Xordan> The problem with altering regions to behave in the correct way is that we break the behaviour that they have now, which could break someone's project.
Jan 18 14:04:59 <jorrit> ok, agreed with that.
Jan 18 14:05:19 <jorrit> but?v Jan 18 14:05:23 <Xordan> otherwise we could just alter the unloading behaviour of current regions and it'd be the same
Jan 18 14:05:33 <Xordan> (or close enough)
Jan 18 14:05:37 * jorrit still doesn't know how Xordan's proposal solves the problems.
Jan 18 14:05:43 <jorrit> i.e. what is the fundamental difference with how regions work now.
Jan 18 14:07:29 <Xordan> regions we have now assume that they're the only region using objects. These assume that they're one of many things holding references to objects, and more than that are only there to keep a hold of enough references so that running unload helper functions on the engine's lists don't remove more than is needed.
Jan 18 14:12:59 <Xordan> So the fundamental difference is in the way we can treat them. With regions we can't say "I am 100% sure that anything not used by another object or collection will be correctly freed by the engine without me doing anything." and definitely not "If a region holds a reference to an object, I know it won't be removed."
Jan 18 14:13:56 <jorrit> Ok. I see.
Jan 18 14:14:12 <jorrit> But I'm not sure that it is a good idea to use that tree structure.
Jan 18 14:14:15 <jorrit> That will only complicate matters.
Jan 18 14:14:24 <jorrit> Especially if you need to find out if something is inside a collection.
Jan 18 14:14:28 <jorrit> You'll have to traverse some tree.
Jan 18 14:14:51 <Xordan> The latter is the bigger one, because right now a region can hold a reference to an object even if it isn't in use (and that happens every time you load an object into a region). With this, you have the guarentee that it won't have a reference held if it isn't being used.
Jan 18 14:15:06 <Xordan> Well, I had in mind that we could hold weak references
Jan 18 14:15:09 <jorrit> Well that's a problem.
Jan 18 14:15:16 <jorrit> Sometimes I *want* objects to stay in memory even if not used.
Jan 18 14:15:24 <jorrit> For examples, factories and materials that I preload.
Jan 18 14:15:26 <jorrit> And that are used later.
Jan 18 14:15:45 <Xordan> In which case we have a few options;
Jan 18 14:15:47 <jorrit> I do that in Crystal Core for example.
Jan 18 14:16:29 <Xordan> 1) Have multiple top-level objects. 2) I mention the option to tell the collection to hold a reference to everything.
Jan 18 14:16:47 <jorrit> But why not have the collection keep a reference in any case?
Jan 18 14:16:50 <jorrit> What would be the problem with that?
Jan 18 14:17:20 <Xordan> How do we know if it's really supposed to be there?
Jan 18 14:17:43 <jorrit> Well if a collection contains it and the collection is in memory then it should be there IMHO.
Jan 18 14:19:11 <Xordan> Then we have a problem in working out what exactly can and can't be unloaded.
Jan 18 14:19:19 <Xordan> (which is where we're at today)
Jan 18 14:19:48 <jorrit> It is possible that the only real good solution is to have a dual reference counting system.
Jan 18 14:20:00 <jorrit> Where you have a 'usage' counter independent from the ref count.
Jan 18 14:21:20 <jorrit> Xordan: I find it rather ad-hoc or arbitrary that a collection should sometimes hold references and sometimes not. That feels clumsy.
Jan 18 14:21:28 <jorrit> Xordan: but I admit it is hard to find other solutions that are not too complicated.
Jan 18 14:21:32 <jorrit> But I want to think about this more.
Jan 18 14:57:33 <res2k> jorrit, Xordan: I don't see why refcount can't be "usage count". Xordan - you right now have a rather human definition of "used", it seems, something along the lines of "a level somewhere needs it". but actually, something is used when it's references. so if jorrit wants to preload stuff he better make sure it's used somehow - for example, by putting it into a "preload region".
Jan 18 14:58:12 <Xordan> Yes
Jan 18 14:59:12 <jorrit> Yes, that might be an option.
Jan 18 14:59:23 <jorrit> But the problem is that that is pretty hard to do in a backwards compatible manner.
Jan 18 14:59:33 <jorrit> As external projects like CEL will not create such 'preload region' right now.
Jan 18 15:00:10 <res2k> what about an "implicit default region"?
Jan 18 15:00:30 <jorrit> res2k: yes but that means that the default behaviour will be the 'leaky' one.
Jan 18 15:00:40 <Xordan> Well, we don't 'have' to deprecate regions right away if we implement a new version of it
Jan 18 15:00:51 <jorrit> Xordan: problem is that CEL doesn't even use regions for those things.
Jan 18 15:00:58 <jorrit> Xordan: so how will you discover that they are not to be removed?
Jan 18 15:00:59 <Xordan> ah hm
Jan 18 15:01:29 <res2k> hmmmm
Jan 18 15:01:54 <res2k> like, currently CEL loads some stuff and expects that to be in the engine lists only?
Jan 18 15:02:00 <res2k> how does it clean out stuff?
Jan 18 15:02:06 <Xordan> Well, that's the reason why I left the possibility not to use collections at all. That means that cel will have to manage unloading itself.
Jan 18 15:02:18 <Xordan> (which isn't nice imo, but it would be there)
Jan 18 15:02:21 <jorrit> res2k: it typically doesn't. These kinds of things are there for the lifetime of the game.
Jan 18 15:02:38 <res2k> so it's *already* leaky :P
Jan 18 15:02:41 <Xordan> :)
Jan 18 15:02:44 <jorrit> Yes but it wants to be leaky.
Jan 18 15:02:50 <jorrit> And it doesn't want something getting rid of those leaks.
Jan 18 15:03:04 <jorrit> Which is what would be the default behaviour if something is not put in a region or collection.
Jan 18 15:03:13 <res2k> what about an "implicit default region"?
Jan 18 15:03:13 <Xordan> Why would it?
Jan 18 15:03:23 <jorrit> Xordan: well if nothing has a reference then why would it stay in memory?
Jan 18 15:03:38 <jorrit> res2k: that's ok. But that means the default behaviour is to leak then.
Jan 18 15:03:47 <Xordan> Because unless you explicitly say to use a collection there will never be any unload
Jan 18 15:04:00 <res2k> jorrit: which is the same as the current behaviour - I don't see the problem?
Jan 18 15:04:09 <jorrit> res2k: I thought Xordan wanted to get rid of that.
Jan 18 15:04:20 <res2k> yes
Jan 18 15:04:21 <res2k> but
Jan 18 15:04:27 <jorrit> Xordan: but how would you say that?
Jan 18 15:05:11 <Xordan> jorrit: Just never use them at all (limiting though).
Jan 18 15:05:12 <res2k> I just assume he's fine with people using explicit regions
Jan 18 15:05:20 <res2k> or manually "clean" the default region if they don't want things to hand around
Jan 18 15:05:23 <res2k> hang*
Jan 18 15:05:43 <jorrit> Xordan: I mean how would the user indicates he wants to use a collection?
Jan 18 15:06:28 <Xordan> Create one, pass a pointer to the load functions. The same way a user indicates he wants to use a region today
Jan 18 15:06:39 <Xordan> (or similar)
Jan 18 15:06:50 <jorrit> ok but...
Jan 18 15:07:02 <jorrit> Hold on, let me quote something:
Jan 18 15:07:19 <jorrit> <res2k> jorrit, Xordan: I don't see why refcount can't be "usage count". Xordan - you right now have a rather human definition of "used", it seems, something along the lines of "a level somewhere needs it". but actually, something is used when it's references. so if jorrit wants to preload stuff he better make sure it's used somehow - for example, by putting it into a "preload region".
Jan 18 15:07:19 <jorrit> <Xordan> Yes
Jan 18 15:07:19 <jorrit> <jorrit> Yes, that might be an option.
Jan 18 15:07:29 <jorrit> Assuming you want to use collections.
Jan 18 15:07:37 <jorrit> And assume you want to get rid of stuff that is not referenced.
Jan 18 15:07:40 <jorrit> How would you go about that?
Jan 18 15:07:53 <jorrit> You don't want to put that stuff in a collection as that would prevent it from being removed.
Jan 18 15:08:00 <jorrit> But if you don't use a collection a default region will be assumed.
Jan 18 15:08:08 <jorrit> Preventing removal again.
Jan 18 15:09:05 <Xordan> yeah.. a default region kind of prevents the ability to choose not to use them at all.
Jan 18 15:09:15 <jorrit> exactly
Jan 18 15:11:00 <Xordan> I favour a 'you use them always or you don't use them at all' system as it keeps things quite simple.
Jan 18 15:11:24 <jorrit> That would almost imply you have some global flag somewhere indicating their usage.
Jan 18 15:11:47 <jorrit> A flag that would have to default to false so that old way is basically the default.
Jan 18 15:11:47 <Xordan> Well, if you literally don't use them then nothing will be automatically unloaded.
Jan 18 15:11:59 <Xordan> Because the unloading functions will never be called
Jan 18 15:12:08 <jorrit> Xordan: when are the unloading functions called?
Jan 18 15:12:16 <Xordan> when the collection is unloaded
Jan 18 15:12:26 <jorrit> Um
Jan 18 15:12:34 <jorrit> So how would it work to unload stuff that is not in any collection?
Jan 18 15:12:38 <jorrit> Assume that we got rid of regions.
Jan 18 15:12:48 <Xordan> The user calls those functions himself
Jan 18 15:12:56 <Xordan> or decides his own way to handle it
Jan 18 15:13:11 <jorrit> hmm ok
Jan 18 15:13:29 <Xordan> (you can create your own groups of objects and only pass that list to those functions for example)
Jan 18 15:14:35 <res2k> jorrit, Xordan: "It might be a good idea to *always* store things in a collection, either one specified by the user, or an implicit (or default) one - otherwise it gets awkward to define consistent semantics if "object is in some collection" vs "an object is not in some collection"."
Jan 18 15:14:58 <Xordan> res2k: When would that need to be defined though?
Jan 18 15:14:59 <res2k> jorrit, Xordan: see especially the part around "awkward"
Jan 18 15:15:29 <res2k> <jorrit> Assuming you want to use collections.
Jan 18 15:15:30 <res2k> <jorrit> And assume you want to get rid of stuff that is not referenced.
Jan 18 15:15:38 <Xordan> Nothing *needs* to know if an object is held in a collection or not.
Jan 18 15:15:44 <res2k> shouldn't, by definition, "stuff that is not referenced" already be gone?
Jan 18 15:16:03 <jorrit> res2k: well right now the engine has its own references (engine lists). But I assume these would be gone.
Jan 18 15:16:15 <jorrit> Or at least no longer keep real refs.
Jan 18 15:16:18 <res2k> they could be weak refs or so.
Jan 18 15:17:32 <Xordan> Well, I hadn't planned to change how the engine lists reference objects. I think an object should always be removed from the list and then freed, not vice versa
Jan 18 15:17:43 <jorrit> Xordan: you'll have to change that.
Jan 18 15:17:46 <Xordan> Why?
Jan 18 15:17:47 <jorrit> Xordan: they keep a reference now.
Jan 18 15:17:55 <jorrit> Xordan: how are you going to get rid of the object if the engine keeps holding that ref?
Jan 18 15:18:21 <Xordan> If we know that only the engine lists are holding a reference then we just call self-destruct
Jan 18 15:18:31 <jorrit> Xordan: no. That's ugly.
Jan 18 15:18:36 <Xordan> hmm, okay
Jan 18 15:18:39 <jorrit> For now we have a chance to let REAL ref counting do its job correctly.
Jan 18 15:18:43 <jorrit> Please let's take that chance.
Jan 18 15:18:52 <res2k> may I make a suggestions? lets collect common use cases on the wiki page
Jan 18 15:18:54 <jorrit> And not have to depend on clumsy checks to see if there are refs remaining.
Jan 18 15:19:23 <res2k> e.g., emulation of the current "leaky" behaviour
Jan 18 15:19:38 <res2k> what Xordan wants, ie fine-grained unloading
Jan 18 15:19:39 <Xordan> In which case we don't really need to call as many helper functions to unload, ref counting will do most of it
Jan 18 15:19:50 <jorrit> Xordan: s/most/all
Jan 18 15:19:51 <res2k> a mix, unloading + having some stuff preloaded ...
Jan 18 15:20:14 <Xordan> We just need to make sure that we have no pointers left over in lists (or will ref counting handle the removal of those too?)
Jan 18 15:20:23 <res2k> weak refs can
Jan 18 15:20:29 <jorrit> Yes with weak refs.
Jan 18 15:20:34 <Xordan> Okay, I didn't know that.
Jan 18 15:21:21 <res2k> or also "manual" removal - e.g. what SelfDestruct does now can be moved into the dtor if ref counting is done properly
Jan 18 15:21:35 <Xordan> Changing the lists to hold weak references means that we will force the user to hold at least one reference of their own for every object they load. So it's quite breaking.
Jan 18 15:21:40 <Xordan> (so a 2.0 thing)
Jan 18 15:21:49 <jorrit> Xordan: not if we have the default region/collection.
Jan 18 15:21:54 <res2k> yeah
Jan 18 15:21:56 <Xordan> Ah, yes