Thinking about lightweight formats that can be used for avatars and looping animations -02

Continuation of the previous post

https://qiita.com/platoronical/items/650a5a34c0e4999e8791

I don't think I've been able to convey what I want to do, but I'll continue for a little while longer

Extending the JSON from the previous post

“canvasWidth”: 4096, // texture width size
“canvasHeight”: 4096, // texture height
“layerCount”: 2,
“baseCanvasWidth”: 1920, // render canvas size
“baseCanvasHeight”: 1080, // render height
“TextureNum”: [ // how many textures to use
{
“imageCount”: 1,
“texImg”: [“texture_room_4096.png”]
}
],
“layers”: [
{
“basefileName”: “01.png”,
“imgType”: “Texture”, // Texture or Sprite
“assignID”: “layer_1”, // Reference name
“animID”:“”, 
“x”: 0,
“y“: 0,
”width“: 1920,
”height“: 1080,
”basePosition_x“: 0,
”basePosition_y“: 0,
”textureZIndex“: 0
}
],
”sprites“:[ //Sprite animation definition
{
”anim01“:[  //Animation
{
”fps":8,
“loop“: 1, // 0 is no loop, 1 is loop
”useTex“:[
”s_01“,”s_02“,”s_03“,”s_04“,”s_05“,”s_06“,”s_07“,”s_08“,”s_09“,”s_10"
]
}

]
}
]
}

Added specifications

• Defined canvas size and rendering area
• Allows multiple textures
• Allows definition of Sprite animation
• fps
• Loop
• Specify textures to use in an array

JSON sample

{
“canvasWidth”: 4096,
“canvasHeight”: 4096,
“layerCount”: 2,
“baseCanvasWidth”: 1920,
“baseCanvasHeight”: 1080,
“TextureNum”: [
{
“imageCount”: 1,
“texImg“: [”texture_room_4096.png“]
}
],

”layers“: [
{
”fileName“: ‘01.png’,
”imgType“: ‘Texture’,
”assignID“: ‘layer_1’,
”animID“:”“,
”x“: 0,
”y“: 0,
”width": 1920,
“height“: 1080,
”basePosition_x“: 0,
”basePosition_y“: 0,
”textureZIndex“: 0
},
{
”fileName“: ‘02.png’,
”imgType“: ‘Sprite’,
”assignID“: ‘s_01’,
”animID“:”anim01“,
”x": 0,
“y“: 1080,
”width“: 408,
”height“: 336,
”basePosition_x“: 415,
”basePosition_y“: 563,
”textureZIndex“: 1
},
{
”fileName“: ‘02.png’,
”imgType": ‘Sprite’,
“assignID“: ‘s_02’,
”animID“:”anim01“,
”x“: 408,
”y“: 1080,
”width“: 408,
”height“: 336,
”basePosition_x“: 415,
”basePosition_y“: 563,
”textureZIndex": 1
}
],

“sprites“:[
{
”anim01“:[
{
”fps“:8,
”loop“: 1,
”useTex“:[
”s_01“,”s_02“,”s_03“,”s_04“,”s_05“,”s_06“,”s_07“,”s_08“,”s_09“,”s_10”

]
}
]
}
]
}

Since rendering didn't work well with pygame, I'm rendering with canvas this time

The animation is played back partially using a single texture

Sorry for using a rather large image (and for the confusing sample).
There is plenty of room in the original texture, so it is possible to animate in several places.

There are three main objectives

• To make it beautiful, it has to be large
• It's hard to make and hard to learn
• The specifications are unclear

I'm creating this prototype to solve the above issues
(I'm also putting the implementation JS below, but the main part is JSON and texture files)

To make it pretty, it has to be big

What gets bigger is the file size and rendering cost
When I thought about implementing lo-fi girl, I thought that there was no format dedicated to looping animation
You could use APNG or GIF, but even in this demo, you need to prepare 10 PNGs of the same size and compress them
Also, as I'll point out next, there is a big issue that you can't embed interactive elements if you make it into a single video file
Sprite animation is an approach that has been around for a long time, but I think it's a good solution to the problem.
The problem of image quality deteriorating during compression is an unavoidable issue with video.

It's hard to make and hard to learn

The learning cost of tools for creating video and avatars is high.
I think it would be advantageous for content creators if it didn't become fat as middleware.

The specifications are unclear

The specifications should be clear.
Both Flash and MMD have a history of working against them due to their black box nature.
At the very least, it would be better for everyone if it was in a form that anyone could touch, even in the indie scene.

The document ChatGPT wrote for me

JSON data documentation

This document explains the data structure of room_texture.model.json and how to use it.

---

1. Overview of JSON data

This JSON file is used to manage 2D graphic layers and set up sprite animations.

• Texture
• Data for drawing static images on a canvas.
• Sprite
• A layer with animation, switching between different images for each frame.
• Texture information
• Information about the texture file to be used.
• Sprite animation settings
• Including frame rate and loop settings.

---

2. JSON structure

{
“canvasWidth”: 4096,
“canvasHeight”: 4096,
“layerCount”: 2,
“baseCanvasWidth”: 1920,
“baseCanvasHeight“: 1080,
”TextureNum“: [
{
”imageCount“: 1,
”texImg“: [”texture_room_4096.png“]
}
],
”layers“: [...],
”sprites": [...]
}

---

3. Details of each field

(1) canvasWidth / canvasHeight

“canvasWidth”: 4096,
“canvasHeight”: 4096,

• Specify the overall canvas size
• This value is different from the normal drawing area, and is the size of the original texture
• Use baseCanvasWidth / baseCanvasHeight for actual drawing

(2) baseCanvasWidth / baseCanvasHeight

“baseCanvasWidth”: 1920,
“baseCanvasHeight”: 1080,

• The canvas size in the first JSON is applied
• The second and subsequent JSONs are scaled down in proportion to baseCanvasWidth

(3) TextureNum (texture information)

“TextureNum”: [
{
“imageCount”: 1,
“texImg”: [“texture_room_4096.png”]
}
]

• Specify the image file name of the texture in texImg
• If there are multiple textures, they can be managed using imageCount

(4) layers (layer information)

Specify the image placement and type for each layer.

“layers”: [
{
“imgType”: “Texture”,
“assignID”: “layer_1”,
“animID”: “”,
“x”: 0, “y”: 0, “width”: 1920, “height”: 1080,
“basePosition_x“: 0, ‘basePosition_y’: 0,
”textureZIndex“: 0
},
{
”imgType“: ‘Sprite’,
”assignID“: ‘s_01’,
”animID“: ‘anim01’,
”x": 0, ‘y’: 1080, ‘width’: 408, ‘height’: 336,
“basePosition_x“: 415, ‘basePosition_y’: 563,
”textureZIndex“: 1
}
]

Details of each field
| Field name | Description |
|-------------|------|
| imgType | ”Texture“ (static image) or ”Sprite" (animation) |
| assignID | Unique ID (also used for sprite animations) |
| animID | Animation ID if the sprite is animated |
| x, y | Coordinates within the texture (crop the image from this range) |
| width, height | Size of the crop area |
| basePosition_x, basePosition_y | Drawing position on canvas |
| textureZIndex | Drawing order (larger value means closer to the front) |

---

(5) sprites (sprite animation settings)

“sprites”: [
{
“anim01”: [
{
“fps”: 8,
“loop“: 1,
”useTex“: [
”s_01“, ‘s_02’, ‘s_03’, ‘s_04’,
”s_05“, ‘s_06’, ‘s_07’, ‘s_08’,
”s_09”, ‘s_10’
]
}
]
}
]

Details of each field
| Field name | Description |
|-------------|------|
| animID | Sprite animation ID (anim01) |
| fps | How many times per second to update the frame |
| loop | 1 → Loop, 0 → Play only once |
| useTex | Specify assignID to determine the order of the animated images |

---

4. How to use

(1) Load JSON

fetch('room_texture.model.json')
.then(response => response.json())
.then(jsonData => addJsonToCanvas(jsonData));

(2) Render to the canvas

function renderCanvas() {
const canvas = document.getElementById('textureCanvas');
const ctx = canvas.getContext('2d');

jsonData.layers.forEach(layer => {
const textureImage = new Image();
textureImage.src = “texture_room_4096.png”;

textureImage.onload = () => {
ctx.drawImage(
textureImage,
layer.x, layer.y, layer.width, layer.height,
layer.basePosition_x, layer.basePosition_y, layer.width, layer.height
);
};
});
}

(3) Sprite animation

function startSpriteAnimation(ctx, jsonData, layer, textureImage) {
const spriteConfig = jsonData.sprites[0][layer.animID][0];

function animateSprite() {
const frameIndex = animationFrames[layer.assignID] % spriteConfig.useTex.length;
const frameAssignID = spriteConfig.useTex[frameIndex];
const frameLayer = jsonData.layers.find(l => l.assignID === frameAssignID);

if (frameLayer) {
ctx.drawImage(
textureImage,
frameLayer.x, frameLayer.y, frameLayer.width, frameLayer.height,
frameLayer.basePosition_x, frameLayer.basePosition_y, frameLayer.width, frameLayer.height
);
}

animationFrames[layer.assignID]++;
if (spriteConfig.loop === 1) {
setTimeout(animateSprite, 1000 / spriteConfig.fps);
}
}

animateSprite();
}

---

5. Summary

• Specify texture information (TextureNum)
• Set layer information in layers
• “Texture” is a static image
• “Sprite” is for animation
• Manage sprite animation with sprites
• Control behavior with fps and loop
• Specify frames with useTex

---

Please refer to this document and use JSON data appropriately! 🚀

Next time

I want to think about the structure of JSON a little more and do something about the player part

The material for this room was borrowed from Nico Nico Monz.
https://commons.nicovideo.jp/works/agreement/nc225712
Thank you, Nanbu Yasumi-san!

Extra

js sample

document.getElementById('loadJsonButton').addEventListener('click', () => {
const fileInput = document.getElementById('jsonFileInput');
const file = fileInput.files[0];

if (!file) {
alert('Please select a JSON file.');
return;
}

const reader = new FileReader();
reader.onload = function (event) {
const jsonData = JSON.parse(event.target.result);
addJsonToCanvas(jsonData);
};
reader.readAsText(file);
});

document.getElementById('resetCanvasButton').addEventListener('click', resetCanvas);

let jsonDataList = [];
let baseCanvasSizeSet = false;
let secondLayerOffset = { x: 0, y: 0 };
let secondLayerCanvas = null;
let secondLayerScale = 1;
let animationFrames = {};
let activeAnimations = {};

// Set up slider events
document.getElementById('sliderX').addEventListener('input', (event) => {
secondLayerOffset.x = parseInt(event.target.value);
requestRender();
});
document.getElementById('sliderY').addEventListener('input', (event) => {
secondLayerOffset.y = parseInt(event.target.value);
requestRender();
});

let renderRequested = false;

function requestRender() {
if (!renderRequested) {
renderRequested = true;
setTimeout(() => {
renderCanvas();
renderRequested = false;
}, 50);
}
}

function addJsonToCanvas(jsonData) {
jsonDataList.push(jsonData);

if (!baseCanvasSizeSet) {
// First JSON is drawn immediately
document.getElementById('textureCanvas').width = jsonData.baseCanvasWidth;
document.getElementById('textureCanvas').height = jsonData.baseCanvasHeight;
baseCanvasSizeSet = true;
renderCanvas();
} else {
// The second JSON is drawn on an off-screen canvas and scaled down
secondLayerCanvas = document.createElement('canvas');
secondLayerCanvas.width = jsonData.baseCanvasWidth;
secondLayerCanvas.height = jsonData.baseCanvasHeight;
const secondLayerCtx = secondLayerCanvas.getContext('2d');

const textureInfo = jsonData.TextureNum[0];
const textureImage = new Image();
textureImage.src = textureInfo.texImg[0];

textureImage.onload = () => {
jsonData.layers.forEach(layer => {
secondLayerCtx.drawImage(
textureImage,
layer.x, layer.y, layer.width, layer.height,
layer.basePosition_x, layer.basePosition_y, layer.width, layer.height
);
});

let scaleW = document.getElementById('textureCanvas').width / secondLayerCanvas.width;
let scaleH = document.getElementById('textureCanvas').height / secondLayerCanvas.height;
secondLayerScale = Math.min(scaleW, scaleH);

requestRender();
};
}
}

function renderCanvas() {
const canvas = document.getElementById('textureCanvas');
const ctx = canvas.getContext('2d');
ctx.clearRect(0, 0, canvas.width, canvas.height);

jsonDataList.forEach((jsonData, index) => {
const textureInfo = jsonData.TextureNum[0];
const textureImage = new Image();
textureImage.src = textureInfo.texImg[0];

textureImage.onload = () => {
const sortedLayers = jsonData.layers.sort((a, b) => a.textureZIndex - b.textureZIndex);
sortedLayers.forEach(layer => {
if (index === 0) {
// immediately draw the first JSON layer
ctx.drawImage(
textureImage,
layer.x, layer.y, layer.width, layer.height,
layer.basePosition_x, layer.basePosition_y, layer.width, layer.height
);
} else if (index === 1 && secondLayerCanvas) {
// Second JSON is drawn scaled down
ctx.drawImage(
secondLayerCanvas,
0, 0, secondLayerCanvas.width, secondLayerCanvas.height,
secondLayerOffset.x, secondLayerOffset.y,
secondLayerCanvas.width * secondLayerScale, secondLayerCanvas.height * secondLayerScale
);
}
});

// Start sprite animation
jsonData.layers.forEach(layer => {
if (layer.imgType === “Sprite”) {
startSpriteAnimation(ctx, jsonData, layer, textureImage);
}
});
};

textureImage.onerror = () => {
console.error(“Failed to load texture image:”, textureImage.src);
};
});
}

function startSpriteAnimation(ctx, jsonData, layer, textureImage) {
const spriteConfig = jsonData.sprites[0][layer.animID][0];
if (!spriteConfig) return;

animationFrames[layer.assignID] = 0;
activeAnimations[layer.assignID] = true;

function animateSprite() {
if (!activeAnimations[layer.assignID]) return;

const frameIndex = animationFrames[layer.assignID] % spriteConfig.useTex.length;
const frameAssignID = spriteConfig.useTex[frameIndex];
const frameLayer = jsonData.layers.find(l => l.assignID === frameAssignID);

if (frameLayer) {
ctx.clearRect(frameLayer.basePosition_x, frameLayer.basePosition_y, frameLayer.width, frameLayer.height);
ctx.drawImage(
textureImage,
frameLayer.x, frameLayer.y, frameLayer.width, frameLayer.height,
frameLayer.basePosition_x, frameLayer.basePosition_y, frameLayer.width, frameLayer.height
);
}

document.getElementById('updateLog').textContent = `Animating ${layer.assignID}: Frame ${frameIndex + 1}`;

animationFrames[layer.assignID]++;
if (spriteConfig.loop === 1 || animationFrames[layer.assignID] < spriteConfig.useTex.length) {
setTimeout(animateSprite, 1000 / spriteConfig.fps);
} else {
activeAnimations[layer.assignID] = false;
}
}

animateSprite();
}

function resetCanvas() {
jsonDataList = [];
baseCanvasSizeSet = false;
secondLayerOffset = { x: 0, y: 0 };
secondLayerCanvas = null;
secondLayerScale = 1;
animationFrames = {};
activeAnimations = {};
document.getElementById('textureCanvas').width = 0;
document.getElementById('textureCanvas').height = 0;
document.getElementById('sliderX').value = 0;
document.getElementById('sliderY').value = 0;
renderCanvas();
}

You can load the second json, but if you do that with a canvas, it will break, so it's something to think about

Loading the second image