big file storage
This commit is contained in:
1 parent
91fe0211d4
commit
9b11a4ded6
1265 files changed
+149378
-60
No files matched your search
@@ -0,0 +1 @@
|
||||
[IMG_6289](https://cloudsfactory.net/wp/wp-content/uploads/2014/03/IMG_6289.jpg)
|
||||
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,2 @@
|
||||
ansio cutting mat
|
||||
juki machines
|
||||
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
for real his life
|
||||
BIN
Binary file not shown.
@@ -0,0 +1,13 @@
|
||||
FROM node:20
|
||||
|
||||
WORKDIR /opt/mcq-test-results
|
||||
COPY . ./
|
||||
|
||||
# ensure our sqlite dir exists
|
||||
RUN mkdir -p /opt/mcq-test-results/db
|
||||
|
||||
# both do the same but always good for logging libraries etc.
|
||||
ENV NODE_ENV=production
|
||||
RUN npm i --omit=dev
|
||||
|
||||
CMD ["npm", "start"]
|
||||
@@ -0,0 +1,89 @@
|
||||
# `mcq-test-results-js`
|
||||
|
||||
I had originally thought to write this service using Rust and Postgres as the data store,
|
||||
but after some consideration I decided to use JavaScript (not TypeScript) to iterate quickly,
|
||||
spend less time fighting with the borrow checker or tweaking `moduleResolution` in a `tsconfig.json`
|
||||
and focus on aspects I considered to be important:
|
||||
|
||||
- clean modular code
|
||||
- dependency injection
|
||||
- easily extensible integration tests
|
||||
- See `src/api.test.js`
|
||||
|
||||
Plus, it's just a prototype right? :P
|
||||
|
||||
I chose SQLite to simplify the deployment and running of the service, as it can be tricky ensuring that a separate DB service
|
||||
is migrated and running before the service starts trying to connect.
|
||||
Additionally it allows the integration tests to use the database layer without having to run multiple services inside a compose network.
|
||||
Finally I wanted to use SQL features like transactions and check constraints to meet requirements of atomicity due to unpaid AWS bills.
|
||||
|
||||
Regarding the XML parsing, I chose to use a library with a simple API and be quite generous in what I accepted in requests instead of using a schema and instead let the application and database layers enforce missing or invalid fields.
|
||||
|
||||
## Using the Service
|
||||
|
||||
### Starting the Service
|
||||
|
||||
```
|
||||
docker compose up
|
||||
```
|
||||
|
||||
`localhost:4567` is forwarded to the listening service inside docker.
|
||||
|
||||
### Tests
|
||||
|
||||
Running tests requires `NPM` to be installed, if you have it you can simply run
|
||||
|
||||
```
|
||||
npm i
|
||||
npm t
|
||||
```
|
||||
|
||||
This will install the dependencies and run the tests.
|
||||
|
||||
Note that the database on your host machine will not be shared with the instance running inside docker.
|
||||
|
||||
### Deployment
|
||||
|
||||
You can find the built service image with
|
||||
|
||||
```
|
||||
docker images | grep mcq-test-results
|
||||
```
|
||||
|
||||
This image can then be tagged and pushed to the appropriate container registry to be run in ECS, or EKS, or whatever container service it is we use here at Markr...
|
||||
|
||||
**IMPORTANT**
|
||||
|
||||
In order for the service's data to be persisted the directory `/opt/mcq-test-results/db` **must** be mounted as a high write speed persistent block store.
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Migrate to PostgreSQL
|
||||
Postgres has excellent built in aggregate functions such as `percentile_cont` that will likely outperform the application level calculations being done by the `simple-statistics` library used here.
|
||||
|
||||
### Logging
|
||||
The application has very little in the way of logging. Suggest a JSON logging library such as [Bunyan](https://github.com/trentm/node-bunyan) to improve debugging and traceability of the application.
|
||||
|
||||
## Assumptions
|
||||
|
||||
### Student Uniqueness
|
||||
The sample data didn't seem to make a great deal of sense; perhaps our friend at the school board butchered it redacting student PII.
|
||||
|
||||
It contained entries where for example:
|
||||
- the same `student-number` was supplied by students with different names
|
||||
- John John & Jean Stephanie both supplied 2398
|
||||
- students with the same name supplied different `student-numbers`
|
||||
- there are 3 Penelope Graces (!) all supplying different student numbers
|
||||
|
||||
Here I assumed that the students and teacher are smart enough to know their names and student numbers and consider them to be unique individuals
|
||||
based on the combination of those attributes.
|
||||
...then again maybe they're just messing with the teacher...
|
||||
|
||||
### On Selecting the highest score
|
||||
When inserting entries into the database, a row is only inserted or updated if the `marksObtained` are greater than the value
|
||||
of an existing row.
|
||||
I had originally thought to standardise the `marksAvailable` based on the highest value seen in any entry for a given `testId`
|
||||
but I didn't know definitively that a test should all have the same `marksAvailable`. The test data didn't seem to indicate that the machines could submit any entries to the contrary either.
|
||||
|
||||
### Size Limits
|
||||
20MB is assumed to be the maximum test submission size
|
||||
@@ -0,0 +1,96 @@
|
||||
# Markr - marking as a service
|
||||
|
||||
You work for Markr, the latest hot startup disrupting the exam-marking scene. Markr has just raised \$800mil in its Series B off the back of a huge deal with every school system in Europe & North America, and wants to pump this cash into producing some sweet analytics for exam boards to analyse overall student performance. You have been made responsible for building out the data ingestion & processing microservice, and you've been asked to prepare a working prototype to demo at the next board meeting (the board is kind of concerned about how much progress is being made, you see).
|
||||
|
||||
## Problem Description
|
||||
|
||||
Students in the school district take multiple-choice exams using those shaded-bubble forms. When the invigilators collect the papers at the end of a test, they feed them into a super-legacy machine which reads the tests, grades them, formats an XML document containing a set of results, and then sends them to the ingestion microservice via a HTTP POST.
|
||||
|
||||
The format of the document isn't super well defined, because the grading machines were build in the early 90's, never documented, and nobody really quite knows what's going on in there. They're also hooked up to a bunch of other critical systems for the school district, so you figure you're stuck with it. It looks something like this.
|
||||
|
||||
```
|
||||
<?xml version="1.0" encoding="UTF-8" ?>
|
||||
<mcq-test-results>
|
||||
<mcq-test-result scanned-on="2017-01-01T00:00:00Z">
|
||||
<first-name>Jimmmy</first-name>
|
||||
<last-name>Student</last-name>
|
||||
<student-number>99999999</student-number>
|
||||
<test-id>78763</test-id>
|
||||
<summary-marks available="10" obtained="2" />
|
||||
<answer question="1" marks-available="1" marks-awarded="1">A</answer>
|
||||
<answer question="2" marks-available="1" marks-awarded="0">B</answer>
|
||||
<answer question="4 marks-available="1" marks-awarded="1">AC</answer>
|
||||
</mcq-test-result>
|
||||
...more mcq-test-result elements follow...
|
||||
</mcq-test-results>
|
||||
```
|
||||
|
||||
Thankfully, the designers of the original system put in a `<summary-marks>` element to add up all the answer data. You're not quite sure if you can trust it, but your boss told you to go with what's in there for now while you smash out this prototype - so you can safely ignore the `<answer>` elements. You also know you've seen some extra fields floating around in here under some circumstances, but they shouldn't concern you - probably some gunk the reporting team needs in there. Finally, you're tangentially aware that there are some other kinds of XML documents that the grading machines make, so you should try not to get your wires crossed with them.
|
||||
|
||||
"At least it's not SOAP", you quietly whisper.
|
||||
|
||||
(P.S. - one of your buddies works for the local school board and got you a "sample" of some real-world data. It's attached in `sample_results.xml`)
|
||||
|
||||
## Requirements
|
||||
|
||||
At Markr, people don't like to muck around - the most important requirements are therefore at the top of the list.
|
||||
|
||||
- The grading machine will POST these documents to a HTTP endpoint at `/import`. The body of the request will be XML file content. They come with a content-type of `text/xml+markr`. You did suggest SSL, but your boss said something along the lines of "won't be here when it's hacked...", so you guess it's OK to ignore that.
|
||||
- You need to ingest these documents into some kind of persistent storage - Markr has been having some problems paying the AWS bill lately so you need to be ready for your instances to turn off at any time. Nobody from the DevOps team responded to you on Slack when you asked what's used around here, so you should just be able to use whatever you want, right?
|
||||
|
||||
For example, the following HTTP POST request should cause a student's test results to be loaded into Markr's storage.
|
||||
|
||||
```
|
||||
curl -X POST -H 'Content-Type: text/xml+markr' http://localhost:4567/import -d @- <<XML
|
||||
<mcq-test-results>
|
||||
<mcq-test-result scanned-on="2017-12-04T12:12:10+11:00">
|
||||
<first-name>Jane</first-name>
|
||||
<last-name>Austen</last-name>
|
||||
<student-number>521585128</student-number>
|
||||
<test-id>1234</test-id>
|
||||
<summary-marks available="20" obtained="13" />
|
||||
</mcq-test-result>
|
||||
</mcq-test-results>
|
||||
XML
|
||||
```
|
||||
|
||||
- The visualisation team have said they want to query your aggregate data at the endpoint `/results/:test-id/aggregate`, and receive a JSON document with the following fields
|
||||
- `mean` - the mean of the awarded marks
|
||||
- `count` - the number of students who took the test
|
||||
- `p25`, `p50`, `p75` - the 25th percentile, median, and 75th percentile scores
|
||||
|
||||
Note that the visualisation team require these numbers to be expressed as _percentages_ (i.e. as a float from 0 to 100) of the available marks in the test.
|
||||
|
||||
For example, the following HTTP GET request should cause the average results for the previous example's test to be returned.
|
||||
|
||||
```
|
||||
curl http://localhost:4567/results/1234/aggregate
|
||||
{"mean":65.0,"stddev":0.0,"min":65.0,"max":65.0,"p25":65.0,"p50":65.0,"p75":65.0,"count":1}
|
||||
```
|
||||
|
||||
- Sometimes, the paper gets a bit folded up when the invigilators put in the scanner, and some of the questions get covered up (and so the score appears lower than it should). When this happens, they usually just scan it again and move on with life. Therefore, you may see a particular student's submission twice, and it will have a different score - just pick the highest one. You'll also need to do the same with the _available_ marks for the test.
|
||||
- These duplicate documents may come in a single request or in multiple requests.
|
||||
- Sometimes, the machines mess up and post you a document missing some important bits. When this happens, it's important that you reject the _entire_ document with an appropriate HTTP error. This causes the machine to print out the offending document (yes, print, as in, on paper) and some poor work experience kid then enters the whole thing manually. If you've already accepted part of the document, that'll cause some confusion which is _way_ above their paygrade.
|
||||
- Although your boss _said_ this was just a working prototype for the board meeting, you have a sneaking suspicion it'll somehow find its way into production. Therefore, you figure you should at least take a swing at error handling and automated tests. Goodness knows you're not getting paid enough for 100% test coverage, but it's probably best if you at least put some tests around the basics.
|
||||
- The current visualisation solution generates printed & (snail) mailed reports overnight, so the aggregate fetching doesn't need to be fast. However, you heard on the grape vine that part of the big fundraising round is going towards building real-time dashboards to be displayed at City Hall - so it's probably worth having a think about that & writing a few things down even if the prototype implementation you build is a bit slow.
|
||||
|
||||
## How we work at Markr
|
||||
|
||||
You know Markr are big on documentation - so you figure should include a README file that specified
|
||||
|
||||
- Any key assumptions you made about the problem and solution spaces. Your boss _hates_ questions, so you should feel free to make (and document!) whatever assumptions you need in order to get the prototype off the ground
|
||||
- A short description of the approach you took
|
||||
- Anything you'd like to draw particular attention to in your solution
|
||||
- Instructions on how to build/run it (the DevOps team aren't the sharpest tools in the shed, so spell it out for them)
|
||||
|
||||
You also remember a few other things from your induction week -
|
||||
|
||||
- You must **provide a docker-compose file** that runs your service. (meta: this is by far the easiest way for us to run/test your code. Please ask us for help if you've never used Docker before! There's also lots of tutorials online, e.g. https://docs.docker.com/compose/gettingstarted/. If you Google search for "<my favourite framework> docker compose file" you should find some starter guides!)
|
||||
- Markr is a forward looking startup - they use the right tool for the job all the time! You can therefore choose whatever language, technologies and frameworks you need to get this done.
|
||||
- Version control is a _must_ - Ops needs to roll stuff back all the time, so it's kind of important they have the history
|
||||
- Payroll around here is a little strange - they pay you in Coles Myer gift cards instead of actual money for some reason (meta: that means we'll give you a \$200 gift card for your time completing this exercise - it's generally easier than giving you actual money because you don't need to wait for payroll, provide us with an ABN/invoice, etc.)
|
||||
- Finally, you've got a lot on at the moment (you're in charge of the GIF Slack channel this week!) so you're going to try and spend about 2-3 hours on this. They _definitely_ don't pay you overtime at Markr. (Meta: this time expectation is for someone experienced at solving problems like this one - if you're not used to writing web servers, you should allow more time.)
|
||||
|
||||
## So, in conclusion....
|
||||
|
||||
Good luck, and get hacking!
|
||||
@@ -0,0 +1,9 @@
|
||||
<mcq-test-results>
|
||||
<mcq-test-result scanned-on="2017-12-04T12:12:10+11:00">
|
||||
<first-name>Jane</first-name>
|
||||
<last-name>Austen</last-name>
|
||||
<student-number>521585128</student-number>
|
||||
<test-id>1234</test-id>
|
||||
<summary-marks available="20" obtained="13" />
|
||||
</mcq-test-result>
|
||||
</mcq-test-results>
|
||||
@@ -0,0 +1,30 @@
|
||||
<?xml version="1.0" encoding="UTF-8" ?>
|
||||
<mcq-test-results>
|
||||
<mcq-test-result scanned-on="2017-12-04T12:12:10+11:00">
|
||||
<first-name>KJ</first-name>
|
||||
<last-name>Alysander</last-name>
|
||||
<student-number>002299</student-number>
|
||||
<test-id>999999</test-id>
|
||||
<answer question="0" marks-available="1" marks-awarded="1">D</answer>
|
||||
<summary-marks available="1" obtained="0" />
|
||||
</mcq-test-result>
|
||||
<mcq-test-result scanned-on="2017-12-04T12:12:10+11:00">
|
||||
<first-name>KJ</first-name>
|
||||
<last-name>Alysander</last-name>
|
||||
<student-number>002299</student-number>
|
||||
<test-id>999999</test-id>
|
||||
<answer question="0" marks-available="1" marks-awarded="1">D</answer>
|
||||
<answer question="1" marks-available="1" marks-awarded="1">D</answer>
|
||||
<answer question="2" marks-available="3" marks-awarded="1">D</answer>
|
||||
<summary-marks available="3" obtained="3" />
|
||||
</mcq-test-result>
|
||||
<mcq-test-result scanned-on="2017-12-04T12:12:10+11:00">
|
||||
<first-name>KJ</first-name>
|
||||
<last-name>Alysander</last-name>
|
||||
<student-number>002299</student-number>
|
||||
<test-id>999999</test-id>
|
||||
<answer question="0" marks-available="1" marks-awarded="1">D</answer>
|
||||
<answer question="1" marks-available="1" marks-awarded="1">D</answer>
|
||||
<summary-marks available="3" obtained="2" />
|
||||
</mcq-test-result>
|
||||
</mcq-test-results>
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,13 @@
|
||||
version: "3.3"
|
||||
|
||||
services:
|
||||
mcq-test-results:
|
||||
image: mcq-test-results
|
||||
build:
|
||||
context: ./
|
||||
dockerfile: ./Dockerfile
|
||||
|
||||
# allow this port through for the CURL test
|
||||
ports:
|
||||
- "4567:4567"
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
import createApi from './src/api.js';
|
||||
import createDb from './src/sqlite.js';
|
||||
|
||||
const db = await createDb();
|
||||
const api = createApi(db);
|
||||
api.listen();
|
||||
@@ -0,0 +1,26 @@
|
||||
--------------------------------------------------------------------------------
|
||||
-- Up
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
CREATE TABLE results (
|
||||
studentId TEXT NOT NULL,
|
||||
firstName TEXT NOT NULL,
|
||||
lastName TEXT NOT NULL,
|
||||
marksAvailable INT NOT NULL,
|
||||
marksObtained INT NOT NULL,
|
||||
scannedOn TEXT NOT NULL,
|
||||
studentNumber INT NOT NULL,
|
||||
testId INT NOT NULL
|
||||
|
||||
CONSTRAINT marksAvailableValid CHECK (marksObtained <= marksAvailable)
|
||||
);
|
||||
|
||||
CREATE INDEX result_ix_testId ON Results (testId);
|
||||
CREATE UNIQUE INDEX result_ix_studentId_testId on Results (studentId, testId);
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
-- Down
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
DROP TABLE results;
|
||||
DROP INDEX results_ix_testId;
|
||||
+10498
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"name": "mcq-test-results-ts",
|
||||
"version": "1.0.0",
|
||||
"description": "",
|
||||
"type": "module",
|
||||
"main": "index.js",
|
||||
"scripts": {
|
||||
"start": "node index.js",
|
||||
"start:dev": "node-dev --respawn index.js",
|
||||
"test": "node --experimental-vm-modules node_modules/jest/bin/jest.js",
|
||||
"test:watch": "node --experimental-vm-modules node_modules/jest/bin/jest.js --watch --detectOpenHandles"
|
||||
},
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"@koa/router": "^12.0.1",
|
||||
"bunyan": "^1.8.15",
|
||||
"koa": "^2.15.0",
|
||||
"koa-body": "^6.0.1",
|
||||
"lodash": "^4.17.21",
|
||||
"simple-statistics": "^7.8.3",
|
||||
"sqlite": "^5.1.1",
|
||||
"sqlite3": "^5.1.7-rc.0",
|
||||
"xml2js": "^0.6.2"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/xml2js": "^0.4.14",
|
||||
"axios": "^1.6.3",
|
||||
"jest": "^29.7.0",
|
||||
"node-dev": "^8.0.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
import koa from 'koa';
|
||||
import koaRouter from '@koa/router'
|
||||
import { koaBody } from 'koa-body';
|
||||
import _ from 'lodash';
|
||||
import * as ss from 'simple-statistics';
|
||||
|
||||
import * as parsing from './parsing.js';
|
||||
import * as sqlite from './sqlite.js';
|
||||
|
||||
export default function createApi(db) {
|
||||
const PORT = 4567;
|
||||
const app = new koa();
|
||||
const router = koaRouter();
|
||||
|
||||
function listen() {
|
||||
const server = app.listen(PORT);
|
||||
console.log('listening on', PORT);
|
||||
return server;
|
||||
}
|
||||
|
||||
router.get('/health', async function health(ctx) {
|
||||
ctx.status = 200;
|
||||
});
|
||||
|
||||
router.post('/import', koaBody({ textLimit: "20mb" }), async function xmlImport(ctx) {
|
||||
try {
|
||||
const parsed = await parsing.parse(ctx.request.body);
|
||||
await db.insertResults(parsed);
|
||||
} catch (e) {
|
||||
console.error(e);
|
||||
return ctx.status = 400;
|
||||
}
|
||||
ctx.status = 200;
|
||||
});
|
||||
|
||||
router.get('/results/:testId/aggregate', async function aggregate(ctx) {
|
||||
const testId = ctx.request.params.testId;
|
||||
ctx.assert(testId, 401, "testId not supplied");
|
||||
const results = await db.getTestResults(testId)
|
||||
ctx.assert(results.length > 0, 404, "test results not found");
|
||||
|
||||
// convert scores to pcts
|
||||
const scores = results.map(r => (r.marksObtained / r.marksAvailable) * 100);
|
||||
|
||||
const stats = {
|
||||
mean: ss.mean(scores),
|
||||
stddev: ss.standardDeviation(scores),
|
||||
min: ss.min(scores),
|
||||
max: ss.max(scores),
|
||||
p25: ss.quantile(scores, 0.25),
|
||||
p50: ss.quantile(scores, 0.50),
|
||||
p75: ss.quantile(scores, 0.75),
|
||||
count: scores.length,
|
||||
};
|
||||
ctx.status = 200;
|
||||
ctx.body = stats;
|
||||
});
|
||||
|
||||
app.use(router.routes());
|
||||
|
||||
return {
|
||||
app,
|
||||
listen,
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
import assert from 'assert/strict';
|
||||
import axios from 'axios';
|
||||
import fs from 'fs/promises';
|
||||
|
||||
import createApi from './api.js';
|
||||
import createDb from './sqlite.js'
|
||||
|
||||
const api = axios.create({
|
||||
baseURL: 'http://localhost:4567',
|
||||
});
|
||||
|
||||
let server;
|
||||
let db;
|
||||
|
||||
beforeAll(async () => {
|
||||
db = await createDb();
|
||||
server = createApi(db).listen();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
server.close();
|
||||
db.db.close();
|
||||
});
|
||||
|
||||
describe('API', function apiTests() {
|
||||
it('listens', async function listenTest() {
|
||||
const res = await api.get('/health');
|
||||
assert.equal(res.status, 200);
|
||||
});
|
||||
|
||||
it('post XML', async function postXML() {
|
||||
const SAMPLE_XML = await fs.readFile('./data/api-test-data.xml')
|
||||
const { data } = await api.post(
|
||||
'/import',
|
||||
SAMPLE_XML,
|
||||
{ headers: { 'Content-Type': 'text/xml+markr' }
|
||||
});
|
||||
});
|
||||
|
||||
it('sample results', async function postXML() {
|
||||
const SAMPLE_XML = await fs.readFile('./data/sample-results.xml')
|
||||
const { data } = await api.post(
|
||||
'/import',
|
||||
SAMPLE_XML,
|
||||
{ headers: { 'Content-Type': 'text/xml+markr' }
|
||||
});
|
||||
});
|
||||
|
||||
it('aggregate json', async function aggregate() {
|
||||
const SAMPLE_XML = await fs.readFile('./data/api-test-data.xml')
|
||||
await api.post(
|
||||
'/import',
|
||||
SAMPLE_XML,
|
||||
{ headers: { 'Content-Type': 'text/xml+markr' }
|
||||
});
|
||||
|
||||
const { data } = await api.get('/results/1234/aggregate');
|
||||
const aggregateData = {
|
||||
mean: 65.0,
|
||||
stddev: 0.0,
|
||||
min: 65.0,
|
||||
max: 65.0,
|
||||
p25: 65.0,
|
||||
p50: 65.0,
|
||||
p75: 65.0,
|
||||
count: 1,
|
||||
};
|
||||
|
||||
assert.deepEqual(data, aggregateData);
|
||||
});
|
||||
|
||||
it('duplicate entries', async function postXML() {
|
||||
const SAMPLE_XML = await fs.readFile('./data/duplicate-student.xml')
|
||||
await api.post(
|
||||
'/import',
|
||||
SAMPLE_XML,
|
||||
{ headers: { 'Content-Type': 'text/xml+markr' }
|
||||
});
|
||||
|
||||
const { data } = await api.get('/results/999999/aggregate');
|
||||
const aggregateData = {
|
||||
mean: 100,
|
||||
stddev: 0.0,
|
||||
min: 100,
|
||||
max: 100,
|
||||
p25: 100,
|
||||
p50: 100,
|
||||
p75: 100,
|
||||
count: 1,
|
||||
};
|
||||
|
||||
// KJ aced it
|
||||
assert.deepEqual(data, aggregateData);
|
||||
});
|
||||
|
||||
it('invalid samples', async function postXML() {
|
||||
assert.rejects(() => api.post(
|
||||
'/import',
|
||||
'<maybe xml></jks>',
|
||||
{ headers: { 'Content-Type': 'text/xml+markr' }
|
||||
}));
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
import assert from 'assert/strict'
|
||||
import xml2js from 'xml2js';
|
||||
import util from 'util';
|
||||
import _ from 'lodash';
|
||||
|
||||
function objectString(obj) {
|
||||
return util.inspect(obj, false, null);
|
||||
}
|
||||
|
||||
// we are ignoring the `answers` in the scan for the moment
|
||||
// and only using the `summary-marks` as per the requirements
|
||||
function parseTestResult(entry) {
|
||||
const firstName = entry['first-name'][0];
|
||||
const lastName = entry['last-name'][0];
|
||||
const marksAvailable = entry['summary-marks'][0]['$'].available;
|
||||
const marksObtained = entry['summary-marks'][0]['$'].obtained;
|
||||
const scannedOn = new Date(entry['$']['scanned-on']);
|
||||
const studentNumber = entry['student-number'][0];
|
||||
|
||||
const testId = entry['test-id'][0];
|
||||
|
||||
// we must combine these features to uniquely (?) identify students from our input
|
||||
// the sample data has duplicates both by full name and by student number
|
||||
const studentId = studentNumber + firstName + lastName;
|
||||
|
||||
const testResult = {
|
||||
studentId,
|
||||
firstName,
|
||||
lastName,
|
||||
marksAvailable,
|
||||
marksObtained,
|
||||
scannedOn,
|
||||
studentNumber,
|
||||
testId,
|
||||
_rawEntry: entry,
|
||||
};
|
||||
|
||||
Object.entries(testResult).forEach(([k, v]) => {
|
||||
assert(testResult[k] !== null, `Missing value for ${k} in entry ${objectString(entry)}`)
|
||||
})
|
||||
|
||||
return testResult;
|
||||
}
|
||||
|
||||
export async function parse(xmlString) {
|
||||
const parser = xml2js.Parser();
|
||||
|
||||
// automatically parse numbers for values and attributes
|
||||
const options = {
|
||||
valueProcessors: [xml2js.processors.parseNumbers],
|
||||
attrValueProcessors: [xml2js.processors.parseNumbers]
|
||||
};
|
||||
|
||||
const xml = await xml2js.parseStringPromise(xmlString, options);
|
||||
const testResults = xml['mcq-test-results']['mcq-test-result'].map(tr => parseTestResult(tr));
|
||||
return testResults;
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
import assert from 'assert/strict'
|
||||
import fs from 'fs/promises';
|
||||
|
||||
import * as parsing from './parsing.js';
|
||||
|
||||
describe('XML parsing', function mcqTestResults() {
|
||||
it('parses sample data', async function sampleData() {
|
||||
const example = await fs.readFile('./data/sample-results.xml', { encoding: 'utf8' });
|
||||
const studentResults = await parsing.parse(example);
|
||||
assert.equal(studentResults.length, 100);
|
||||
});
|
||||
})
|
||||
@@ -0,0 +1,59 @@
|
||||
import sqlite3 from 'sqlite3'
|
||||
import { open } from 'sqlite'
|
||||
|
||||
export default async function createDb() {
|
||||
sqlite3.verbose()
|
||||
|
||||
const db = await open({
|
||||
filename: './db/test-results.sqlite',
|
||||
driver: sqlite3.Database
|
||||
});
|
||||
|
||||
await db.migrate();
|
||||
|
||||
async function insertResults(results) {
|
||||
await db.run('BEGIN TRANSACTION');
|
||||
|
||||
// our query here prevents rows from being added
|
||||
await Promise.all(results.map(async result => {
|
||||
// template string doesn't work with this lib for some reason...
|
||||
db.run("\
|
||||
INSERT OR REPLACE INTO results \
|
||||
SELECT $studentId, $firstName, $lastName, $marksAvailable, $marksObtained, $scannedOn, $studentNumber, $testId \
|
||||
WHERE NOT EXISTS ( \
|
||||
SELECT 1 \
|
||||
FROM results \
|
||||
WHERE studentId = $studentId \
|
||||
AND testId = $testId \
|
||||
AND marksObtained > $marksObtained \
|
||||
); \
|
||||
", {
|
||||
$firstName: result.firstName,
|
||||
$lastName: result.lastName,
|
||||
$marksAvailable: result.marksAvailable,
|
||||
$marksObtained: result.marksObtained,
|
||||
$scannedOn: result.scannedOn.toString(),
|
||||
$studentNumber: result.studentNumber,
|
||||
$studentId: result.studentId,
|
||||
$testId: result.testId,
|
||||
});
|
||||
}));
|
||||
|
||||
db.run('COMMIT');
|
||||
}
|
||||
|
||||
async function getTestResults(testId) {
|
||||
const results = await db.all(`
|
||||
SELECT * FROM results
|
||||
WHERE testId = $testId;
|
||||
`, { $testId: testId });
|
||||
return results;
|
||||
}
|
||||
|
||||
return {
|
||||
insertResults,
|
||||
getTestResults,
|
||||
|
||||
db,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
![[Pasted image 20260128230423.png]]
|
||||
|
||||
|
||||
![[Pasted image 20260212123646.png]]
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
punk rock spider web
|
||||
## spooky skeleton death stick
|
||||
|
||||
![[Pasted image 20260203215833.jpg]]
|
||||
|
||||
smothered mate
|
||||
HS
|
||||
CC
|
||||
|
||||
super long back rank double rook mate
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in new issue
Block a user