Commit c373ba99 authored by Paul Mundt's avatar Paul Mundt
Browse files
parents 6f3529f0 851b147e
ARM TCM (Tightly-Coupled Memory) handling in Linux
----
Written by Linus Walleij <linus.walleij@stericsson.com>
Some ARM SoC:s have a so-called TCM (Tightly-Coupled Memory).
This is usually just a few (4-64) KiB of RAM inside the ARM
processor.
Due to being embedded inside the CPU The TCM has a
Harvard-architecture, so there is an ITCM (instruction TCM)
and a DTCM (data TCM). The DTCM can not contain any
instructions, but the ITCM can actually contain data.
The size of DTCM or ITCM is minimum 4KiB so the typical
minimum configuration is 4KiB ITCM and 4KiB DTCM.
ARM CPU:s have special registers to read out status, physical
location and size of TCM memories. arch/arm/include/asm/cputype.h
defines a CPUID_TCM register that you can read out from the
system control coprocessor. Documentation from ARM can be found
at http://infocenter.arm.com, search for "TCM Status Register"
to see documents for all CPUs. Reading this register you can
determine if ITCM (bit 0) and/or DTCM (bit 16) is present in the
machine.
There is further a TCM region register (search for "TCM Region
Registers" at the ARM site) that can report and modify the location
size of TCM memories at runtime. This is used to read out and modify
TCM location and size. Notice that this is not a MMU table: you
actually move the physical location of the TCM around. At the
place you put it, it will mask any underlying RAM from the
CPU so it is usually wise not to overlap any physical RAM with
the TCM. The TCM memory exists totally outside the MMU and will
override any MMU mappings.
Code executing inside the ITCM does not "see" any MMU mappings
and e.g. register accesses must be made to physical addresses.
TCM is used for a few things:
- FIQ and other interrupt handlers that need deterministic
timing and cannot wait for cache misses.
- Idle loops where all external RAM is set to self-refresh
retention mode, so only on-chip RAM is accessible by
the CPU and then we hang inside ITCM waiting for an
interrupt.
- Other operations which implies shutting off or reconfiguring
the external RAM controller.
There is an interface for using TCM on the ARM architecture
in <asm/tcm.h>. Using this interface it is possible to:
- Define the physical address and size of ITCM and DTCM.
- Tag functions to be compiled into ITCM.
- Tag data and constants to be allocated to DTCM and ITCM.
- Have the remaining TCM RAM added to a special
allocation pool with gen_pool_create() and gen_pool_add()
and provice tcm_alloc() and tcm_free() for this
memory. Such a heap is great for things like saving
device state when shutting off device power domains.
A machine that has TCM memory shall select HAVE_TCM in
arch/arm/Kconfig for itself, and then the
rest of the functionality will depend on the physical
location and size of ITCM and DTCM to be defined in
mach/memory.h for the machine. Code that needs to use
TCM shall #include <asm/tcm.h> If the TCM is not located
at the place given in memory.h it will be moved using
the TCM Region registers.
Functions to go into itcm can be tagged like this:
int __tcmfunc foo(int bar);
Variables to go into dtcm can be tagged like this:
int __tcmdata foo;
Constants can be tagged like this:
int __tcmconst foo;
To put assembler into TCM just use
.section ".tcm.text" or .section ".tcm.data"
respectively.
Example code:
#include <asm/tcm.h>
/* Uninitialized data */
static u32 __tcmdata tcmvar;
/* Initialized data */
static u32 __tcmdata tcmassigned = 0x2BADBABEU;
/* Constant */
static const u32 __tcmconst tcmconst = 0xCAFEBABEU;
static void __tcmlocalfunc tcm_to_tcm(void)
{
int i;
for (i = 0; i < 100; i++)
tcmvar ++;
}
static void __tcmfunc hello_tcm(void)
{
/* Some abstract code that runs in ITCM */
int i;
for (i = 0; i < 100; i++) {
tcmvar ++;
}
tcm_to_tcm();
}
static void __init test_tcm(void)
{
u32 *tcmem;
int i;
hello_tcm();
printk("Hello TCM executed from ITCM RAM\n");
printk("TCM variable from testrun: %u @ %p\n", tcmvar, &tcmvar);
tcmvar = 0xDEADBEEFU;
printk("TCM variable: 0x%x @ %p\n", tcmvar, &tcmvar);
printk("TCM assigned variable: 0x%x @ %p\n", tcmassigned, &tcmassigned);
printk("TCM constant: 0x%x @ %p\n", tcmconst, &tcmconst);
/* Allocate some TCM memory from the pool */
tcmem = tcm_alloc(20);
if (tcmem) {
printk("TCM Allocated 20 bytes of TCM @ %p\n", tcmem);
tcmem[0] = 0xDEADBEEFU;
tcmem[1] = 0x2BADBABEU;
tcmem[2] = 0xCAFEBABEU;
tcmem[3] = 0xDEADBEEFU;
tcmem[4] = 0x2BADBABEU;
for (i = 0; i < 5; i++)
printk("TCM tcmem[%d] = %08x\n", i, tcmem[i]);
tcm_free(tcmem, 20);
}
}
......@@ -194,7 +194,6 @@ static void cfag12864b_blit(void)
*/
#include <stdio.h>
#include <string.h>
#define EXAMPLES 6
......
......@@ -408,6 +408,26 @@ You can attach the current shell task by echoing 0:
# echo 0 > tasks
2.3 Mounting hierarchies by name
--------------------------------
Passing the name=<x> option when mounting a cgroups hierarchy
associates the given name with the hierarchy. This can be used when
mounting a pre-existing hierarchy, in order to refer to it by name
rather than by its set of active subsystems. Each hierarchy is either
nameless, or has a unique name.
The name should match [\w.-]+
When passing a name=<x> option for a new hierarchy, you need to
specify subsystems manually; the legacy behaviour of mounting all
subsystems when none are explicitly specified is not supported when
you give a subsystem a name.
The name of the subsystem appears as part of the hierarchy description
in /proc/mounts and /proc/<pid>/cgroups.
3. Kernel API
=============
......@@ -501,7 +521,7 @@ rmdir() will fail with it. From this behavior, pre_destroy() can be
called multiple times against a cgroup.
int can_attach(struct cgroup_subsys *ss, struct cgroup *cgrp,
struct task_struct *task)
struct task_struct *task, bool threadgroup)
(cgroup_mutex held by caller)
Called prior to moving a task into a cgroup; if the subsystem
......@@ -509,14 +529,20 @@ returns an error, this will abort the attach operation. If a NULL
task is passed, then a successful result indicates that *any*
unspecified task can be moved into the cgroup. Note that this isn't
called on a fork. If this method returns 0 (success) then this should
remain valid while the caller holds cgroup_mutex.
remain valid while the caller holds cgroup_mutex. If threadgroup is
true, then a successful result indicates that all threads in the given
thread's threadgroup can be moved together.
void attach(struct cgroup_subsys *ss, struct cgroup *cgrp,
struct cgroup *old_cgrp, struct task_struct *task)
struct cgroup *old_cgrp, struct task_struct *task,
bool threadgroup)
(cgroup_mutex held by caller)
Called after the task has been attached to the cgroup, to allow any
post-attachment activity that requires memory allocations or blocking.
If threadgroup is true, the subsystem should take care of all threads
in the specified thread's threadgroup. Currently does not support any
subsystem that might need the old_cgrp for every thread in the group.
void fork(struct cgroup_subsy *ss, struct task_struct *task)
......
......@@ -179,6 +179,9 @@ The reclaim algorithm has not been modified for cgroups, except that
pages that are selected for reclaiming come from the per cgroup LRU
list.
NOTE: Reclaim does not work for the root cgroup, since we cannot set any
limits on the root cgroup.
2. Locking
The memory controller uses the following hierarchy
......@@ -210,6 +213,7 @@ We can alter the memory limit:
NOTE: We can use a suffix (k, K, m, M, g or G) to indicate values in kilo,
mega or gigabytes.
NOTE: We can write "-1" to reset the *.limit_in_bytes(unlimited).
NOTE: We cannot set limits on the root cgroup any more.
# cat /cgroups/0/memory.limit_in_bytes
4194304
......@@ -375,7 +379,42 @@ cgroups created below it.
NOTE2: This feature can be enabled/disabled per subtree.
7. TODO
7. Soft limits
Soft limits allow for greater sharing of memory. The idea behind soft limits
is to allow control groups to use as much of the memory as needed, provided
a. There is no memory contention
b. They do not exceed their hard limit
When the system detects memory contention or low memory control groups
are pushed back to their soft limits. If the soft limit of each control
group is very high, they are pushed back as much as possible to make
sure that one control group does not starve the others of memory.
Please note that soft limits is a best effort feature, it comes with
no guarantees, but it does its best to make sure that when memory is
heavily contended for, memory is allocated based on the soft limit
hints/setup. Currently soft limit based reclaim is setup such that
it gets invoked from balance_pgdat (kswapd).
7.1 Interface
Soft limits can be setup by using the following commands (in this example we
assume a soft limit of 256 megabytes)
# echo 256M > memory.soft_limit_in_bytes
If we want to change this to 1G, we can at any time use
# echo 1G > memory.soft_limit_in_bytes
NOTE1: Soft limits take effect over a long period of time, since they involve
reclaiming memory for balancing between memory cgroups
NOTE2: It is recommended to set the soft limit always below the hard limit,
otherwise the hard limit will take precedence.
8. TODO
1. Add support for accounting huge pages (as a separate controller)
2. Make per-cgroup scanner reclaim not-shared pages first
......
......@@ -54,20 +54,23 @@ features surfaced as a result:
3.1 General format of the API:
struct dma_async_tx_descriptor *
async_<operation>(<op specific parameters>,
enum async_tx_flags flags,
struct dma_async_tx_descriptor *dependency,
dma_async_tx_callback callback_routine,
void *callback_parameter);
async_<operation>(<op specific parameters>, struct async_submit ctl *submit)
3.2 Supported operations:
memcpy - memory copy between a source and a destination buffer
memset - fill a destination buffer with a byte value
xor - xor a series of source buffers and write the result to a
destination buffer
xor_zero_sum - xor a series of source buffers and set a flag if the
result is zero. The implementation attempts to prevent
writes to memory
memcpy - memory copy between a source and a destination buffer
memset - fill a destination buffer with a byte value
xor - xor a series of source buffers and write the result to a
destination buffer
xor_val - xor a series of source buffers and set a flag if the
result is zero. The implementation attempts to prevent
writes to memory
pq - generate the p+q (raid6 syndrome) from a series of source buffers
pq_val - validate that a p and or q buffer are in sync with a given series of
sources
datap - (raid6_datap_recov) recover a raid6 data block and the p block
from the given sources
2data - (raid6_2data_recov) recover 2 raid6 data blocks from the given
sources
3.3 Descriptor management:
The return value is non-NULL and points to a 'descriptor' when the operation
......@@ -80,8 +83,8 @@ acknowledged by the application before the offload engine driver is allowed to
recycle (or free) the descriptor. A descriptor can be acked by one of the
following methods:
1/ setting the ASYNC_TX_ACK flag if no child operations are to be submitted
2/ setting the ASYNC_TX_DEP_ACK flag to acknowledge the parent
descriptor of a new operation.
2/ submitting an unacknowledged descriptor as a dependency to another
async_tx call will implicitly set the acknowledged state.
3/ calling async_tx_ack() on the descriptor.
3.4 When does the operation execute?
......@@ -119,30 +122,42 @@ of an operation.
Perform a xor->copy->xor operation where each operation depends on the
result from the previous operation:
void complete_xor_copy_xor(void *param)
void callback(void *param)
{
printk("complete\n");
struct completion *cmp = param;
complete(cmp);
}
int run_xor_copy_xor(struct page **xor_srcs,
int xor_src_cnt,
struct page *xor_dest,
size_t xor_len,
struct page *copy_src,
struct page *copy_dest,
size_t copy_len)
void run_xor_copy_xor(struct page **xor_srcs,
int xor_src_cnt,
struct page *xor_dest,
size_t xor_len,
struct page *copy_src,
struct page *copy_dest,
size_t copy_len)
{
struct dma_async_tx_descriptor *tx;
addr_conv_t addr_conv[xor_src_cnt];
struct async_submit_ctl submit;
addr_conv_t addr_conv[NDISKS];
struct completion cmp;
init_async_submit(&submit, ASYNC_TX_XOR_DROP_DST, NULL, NULL, NULL,
addr_conv);
tx = async_xor(xor_dest, xor_srcs, 0, xor_src_cnt, xor_len, &submit)
tx = async_xor(xor_dest, xor_srcs, 0, xor_src_cnt, xor_len,
ASYNC_TX_XOR_DROP_DST, NULL, NULL, NULL);
tx = async_memcpy(copy_dest, copy_src, 0, 0, copy_len,
ASYNC_TX_DEP_ACK, tx, NULL, NULL);
tx = async_xor(xor_dest, xor_srcs, 0, xor_src_cnt, xor_len,
ASYNC_TX_XOR_DROP_DST | ASYNC_TX_DEP_ACK | ASYNC_TX_ACK,
tx, complete_xor_copy_xor, NULL);
submit->depend_tx = tx;
tx = async_memcpy(copy_dest, copy_src, 0, 0, copy_len, &submit);
init_completion(&cmp);
init_async_submit(&submit, ASYNC_TX_XOR_DROP_DST | ASYNC_TX_ACK, tx,
callback, &cmp, addr_conv);
tx = async_xor(xor_dest, xor_srcs, 0, xor_src_cnt, xor_len, &submit);
async_tx_issue_pending_all();
wait_for_completion(&cmp);
}
See include/linux/async_tx.h for more information on the flags. See the
......
......@@ -4,7 +4,7 @@ Shared Subtrees
Contents:
1) Overview
2) Features
3) smount command
3) Setting mount states
4) Use-case
5) Detailed semantics
6) Quiz
......@@ -41,14 +41,14 @@ replicas continue to be exactly same.
Here is an example:
Lets say /mnt has a mount that is shared.
Let's say /mnt has a mount that is shared.
mount --make-shared /mnt
note: mount command does not yet support the --make-shared flag.
I have included a small C program which does the same by executing
'smount /mnt shared'
Note: mount(8) command now supports the --make-shared flag,
so the sample 'smount' program is no longer needed and has been
removed.
#mount --bind /mnt /tmp
# mount --bind /mnt /tmp
The above command replicates the mount at /mnt to the mountpoint /tmp
and the contents of both the mounts remain identical.
......@@ -58,8 +58,8 @@ replicas continue to be exactly same.
#ls /tmp
a b c
Now lets say we mount a device at /tmp/a
#mount /dev/sd0 /tmp/a
Now let's say we mount a device at /tmp/a
# mount /dev/sd0 /tmp/a
#ls /tmp/a
t1 t2 t2
......@@ -80,21 +80,20 @@ replicas continue to be exactly same.
Here is an example:
Lets say /mnt has a mount which is shared.
#mount --make-shared /mnt
Let's say /mnt has a mount which is shared.
# mount --make-shared /mnt
Lets bind mount /mnt to /tmp
#mount --bind /mnt /tmp
Let's bind mount /mnt to /tmp
# mount --bind /mnt /tmp
the new mount at /tmp becomes a shared mount and it is a replica of
the mount at /mnt.
Now lets make the mount at /tmp; a slave of /mnt
#mount --make-slave /tmp
[or smount /tmp slave]
Now let's make the mount at /tmp; a slave of /mnt
# mount --make-slave /tmp
lets mount /dev/sd0 on /mnt/a
#mount /dev/sd0 /mnt/a
let's mount /dev/sd0 on /mnt/a
# mount /dev/sd0 /mnt/a
#ls /mnt/a
t1 t2 t3
......@@ -104,9 +103,9 @@ replicas continue to be exactly same.
Note the mount event has propagated to the mount at /tmp
However lets see what happens if we mount something on the mount at /tmp
However let's see what happens if we mount something on the mount at /tmp
#mount /dev/sd1 /tmp/b
# mount /dev/sd1 /tmp/b
#ls /tmp/b
s1 s2 s3
......@@ -124,12 +123,11 @@ replicas continue to be exactly same.
2d) A unbindable mount is a unbindable private mount
lets say we have a mount at /mnt and we make is unbindable
let's say we have a mount at /mnt and we make is unbindable
#mount --make-unbindable /mnt
[ smount /mnt unbindable ]
# mount --make-unbindable /mnt
Lets try to bind mount this mount somewhere else.
Let's try to bind mount this mount somewhere else.
# mount --bind /mnt /tmp
mount: wrong fs type, bad option, bad superblock on /mnt,
or too many mounted file systems
......@@ -137,149 +135,15 @@ replicas continue to be exactly same.
Binding a unbindable mount is a invalid operation.
3) smount command
3) Setting mount states
Currently the mount command is not aware of shared subtree features.
Work is in progress to add the support in mount ( util-linux package ).
Till then use the following program.
The mount command (util-linux package) can be used to set mount
states:
------------------------------------------------------------------------
//
//this code was developed my Miklos Szeredi <miklos@szeredi.hu>
//and modified by Ram Pai <linuxram@us.ibm.com>
// sample usage:
// smount /tmp shared
//
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/mount.h>
#include <sys/fsuid.h>
#ifndef MS_REC
#define MS_REC 0x4000 /* 16384: Recursive loopback */
#endif
#ifndef MS_SHARED
#define MS_SHARED 1<<20 /* Shared */
#endif
#ifndef MS_PRIVATE
#define MS_PRIVATE 1<<18 /* Private */
#endif
#ifndef MS_SLAVE
#define MS_SLAVE 1<<19 /* Slave */
#endif
#ifndef MS_UNBINDABLE
#define MS_UNBINDABLE 1<<17 /* Unbindable */
#endif
int main(int argc, char *argv[])
{
int type;
if(argc != 3) {
fprintf(stderr, "usage: %s dir "
"<rshared|rslave|rprivate|runbindable|shared|slave"
"|private|unbindable>\n" , argv[0]);
return 1;
}
fprintf(stdout, "%s %s %s\n", argv[0], argv[1], argv[2]);
if (strcmp(argv[2],"rshared")==0)
type=(MS_SHARED|MS_REC);
else if (strcmp(argv[2],"rslave")==0)
type=(MS_SLAVE|MS_REC);
else if (strcmp(argv[2],"rprivate")==0)
type=(MS_PRIVATE|MS_REC);
else if (strcmp(argv[2],"runbindable")==0)
type=(MS_UNBINDABLE|MS_REC);
else if (strcmp(argv[2],"shared")==0)
type=MS_SHARED;
else if (strcmp(argv[2],"slave")==0)
type=MS_SLAVE;
else if (strcmp(argv[2],"private")==0)
type=MS_PRIVATE;
else if (strcmp(argv[2],"unbindable")==0)
type=MS_UNBINDABLE;
else {
fprintf(stderr, "invalid operation: %s\n", argv[2]);
return 1;
}
setfsuid(getuid());
if(mount("", argv[1], "dontcare", type, "") == -1) {
perror("mount");
return 1;
}
return 0;
}
-----------------------------------------------------------------------
Copy the above code snippet into smount.c
gcc -o smount smount.c
(i) To mark all the mounts under /mnt as shared execute the following
command:
smount /mnt rshared
the corresponding syntax planned for mount command is
mount --make-rshared /mnt
just to mark a mount /mnt as shared, execute the following
command:
smount /mnt shared
the corresponding syntax planned for mount command is
mount --make-shared /mnt
(ii) To mark all the shared mounts under /mnt as slave execute the
following
command:
smount /mnt rslave
the corresponding syntax planned for mount command is
mount --make-rslave /mnt
just to mark a mount /mnt as slave, execute the following
command:
smount /mnt slave
the corresponding syntax planned for mount command is
mount --make-slave /mnt
(iii) To mark all the mounts under /mnt as private execute the
following command:
smount /mnt rprivate
the corresponding syntax planned for mount command is
mount --make-rprivate /mnt
just to mark a mount /mnt as private, execute the following
command:
smount /mnt private
the corresponding syntax planned for mount command is
mount --make-private /mnt
NOTE: by default all the mounts are created as private. But if
you want to change some shared/slave/unbindable mount as
private at a later point in time, this command can help.
(iv) To mark all the mounts under /mnt as unbindable execute the
following
command:
smount /mnt runbindable
the corresponding syntax planned for mount command is
mount --make-runbindable /mnt
just to mark a mount /mnt as unbindable, execute the following
command:
smount /mnt unbindable
the corresponding syntax planned for mount command is
mount --make-unbindable /mnt
mount --make-shared mountpoint
mount --make-slave mountpoint
mount --make-private mountpoint
mount --make-unbindable mountpoint
4) Use cases
......@@ -350,7 +214,7 @@ replicas continue to be exactly same.
mount --rbind / /view/v3
mount --rbind / /view/v4
and if /usr has a versioning filesystem mounted, than that
and if /usr has a versioning filesystem mounted, then that
mount appears at /view/v1/usr, /view/v2/usr, /view/v3/usr and
/view/v4/usr too
......@@ -390,7 +254,7 @@ replicas continue to be exactly same.
For example:
mount --make-shared /mnt