US11452077B2ActiveUtilityA1
Wireless network resource allocation method employing generative adversarial reinforcement learning
Est. expiryDec 24, 2039(~13.5 yrs left)· nominal 20-yr term from priority
H04W 72/53G06N 3/047G06N 3/045G06N 3/08G06N 3/094G06N 3/0475G06N 3/092H04W 72/04H04W 28/16
43
PatentIndex Score
0
Cited by
8
References
20
Claims
Abstract
A wireless network resource allocating method comprises: initializing a generator network G and a discriminator network D; performing resource allocation; training weights of the generator network G and the discriminator network D; and implementing wireless network resource allocation.
Claims
exact text as granted — not AI-modifiedWhat is claimed is:
1. A wireless network resource allocating method based on generative adversarial reinforcement learning, wherein a generative adversarial reinforcement learning algorithm comprises two neural networks, which are a generator network G and a discriminator network D, wherein the method comprises:
(1) initializing the generator network G and the discriminator network D, comprising:
(1.1) randomly initializing weights of the generator network G and the discriminator network D through Gaussian distribution; and setting a network Ĝ which has a same structure as that of the generator network G, and initializing a weight of the network Ĝ by copying the weight of the generator network G;
(1.2) inputting a network state s into the generator network G, outputting an N×N-dimensional vector by the generator network G, and sequentially dividing the N a ×N-dimensional vector output by the generator network G into N a N-dimensional vectors;
inputting an N-dimensional vector into the discriminator network D, the N-dimensional vector input into the discriminator network D being taken from an output of the generator network G or being obtained by calculation based on an output of the network Ĝ and an instant return; and
determining by the discriminator network D that the N-dimensional vector input into the discriminator network D is taken from the output of the generator network G if an absolute value of a difference between a scalar and 0 is less than an absolute value of a difference between the scalar and 1, and determining by the discriminator network D that the N-dimensional vector input into the discriminator network D is obtained by calculation based on the output of the network Ĝ and the instant return if the absolute value of the difference between the scalar and 1 is less than the absolute value of the difference between the scalar and 0, where the scalar is output by the generator network G for representing authenticity of an input;
wherein N represents the number of samples sampled from Z(s,a), Z(s,a) represents a cumulative return distribution obtained from an action a under the network state s, the network state s represents the number of requests for each type of service in a time interval, the action a represents a magnitude of a bandwidth allocated for each type of service, N a represents the number of effective actions, and a p th vector in the N a N-dimensional vectors represents sampling values of an overall return distribution obtained from a p th action;
(2) performing resource allocation, comprising:
(2.1) acquiring an observed value s t of the network state s at a current time moment t by a radio resource manager; selecting an action a t using a ϵ-greedy strategy by the radio resource manager; receiving a system return value J by the radio resource manager when the action a t is executed, and observing an observed value s t+1 of the network state s at a time moment t+1;
wherein selecting the action a t using the ϵ-greedy strategy by the radio resource manager comprises:
acquiring a random number from a (0,1) uniform distribution by the radio resource manager;
randomly selecting an effective action by the radio resource manager if the random number is less than ϵ;
inputting the observed value s t into the generator network G by the radio resource manager to obtain sampling values of cumulative return distributions of N a actions if the random number is greater than or equal to ϵ; calculating a mean value of the sampling values of the cumulative return distribution of each action; and selecting an action corresponding to a maximum mean value;
(2.2) setting two thresholds c 1 and c 2 and an absolute value ξ of a fixed instant return by the radio resource manager, where c 1 >c 2 , and setting an instant return r t at the time moment t to be ξ when J>c 1 , to be 0 when c 2 <J<c 1 , and to be −ξ when J<c 2 ;
(2.3) storing quadruples (s t , a t , r t , s t+1 ) by the radio resource manager in a buffer area with a size of N B ; deleting a quadruple earliest stored in the buffer area and storing a newest quadruple into the buffer area when the buffer area is full;
(3) every K times the resource allocation of the step (2) is performed, training the weights of the generator network G and the discriminator network D using the quadruples stored in the buffer area , comprising:
(3.1) training the discriminator network D first, comprising:
randomly selecting m quadruples (s t , a t , r t , s t+1 ) from the buffer area as training data;
combining the observed values s t of the network state at the time moment t in the m quadruples into an m×N s matrix [s 1 , s 2 , . . . s m ] T , where s m represents an m th observed value s t of the network state at the time moment t; inputting the m×N s matrix [s 1 , s 2 , . . . s m ] T into the generator network Ĝ to obtain sampling values of the cumulative return distribution of each action under the m observed value s t of the network state at the time moment t, and retaining sampling values corresponding to a 1 , a 2 , . . . a m , denoted as G(s 1 ), G(s 2 ), . . . G(s m ), where N s represents the number of service types, G(s m ) represents N sampling values of a return obtained by taking the action a m under the m th observed value s t of the network state at the time moment t, which are recorded as sampling values of a distribution of an estimated action value;
combining m observed values s t+1 of the network state at the time moment t+1 in the training data into an m×N s matrix [s 1 ′, s 2 ′, . . . s m ′] T , and inputting the m×N s matrix [s 1 ′, s 2 ′, . . . s m ′] T into the network Ĝ to obtain sampling values of the cumulative return distribution of each action under the m observed value s t+1 of the network state at the time moment t+1, and retaining sampling values of mean values of the maximum cumulative return generated under each observed value s t+1 of the network state at the time moment t+1, which are denoted as Ĝ(s 1 ′), Ĝ(s 2 ′), . . . Ĝ(s m ′), where s m ′ represents a m th observed value s t+1 of the network state at the time moment t+1;
making
y i =r i +γ*Ĝ ( s i ′), i= 1,2, . . . m (1),
where y i represents a sampling value of a distribution of a target action value, r i represents the instant return, and γ represents a discount factor;
randomly acquiring m samples from a (0,1) uniform distribution, denoted as ε 1 , ε 2 , . . . ε m ;
making
{circumflex over (x)} i =ε i *y i +(1−ε i )* G ( s i ), i= 1,2, . . . m (2),
where {circumflex over (x)} i represents a weighted sum of the sampling value of the distribution of the target action value and the sampling value of the distribution of the estimated action value;
wherein a loss function L D of the discriminator network D is:
L
D
=
1
m
∑
i
=
1
m
(
D
(
G
(
s
i
)
)
-
D
(
y
i
)
+
λ
(
∇
x
^
i
D
(
x
^
i
)
2
-
1
)
2
)
(
3
)
where D(G(s)) represents an output of the discriminator network D when an input is G(s i ); D(y i ) represents an output of the discriminator network D when an input is y i ; D({circumflex over (x)} i ) represents an output of the discriminator network D when an input is {circumflex over (x)} i , ∇ {circumflex over (x)} i D({circumflex over (x)} i ) represents a gradient value obtained by derivation of D({circumflex over (x)} i ) with respect to {circumflex over (x)} i , and λ represents a penalty factor, and
training the weight of the discriminator network D using a gradient descent algorithm to complete the training of the discriminator network D for one time;
(3.2) obtaining a newest weight of the discriminator network D to participate in the training of the generator network G, after training the discriminator network D for n d times, wherein a loss function L G of the generator network G is:
L
G
=
-
1
m
∑
i
=
1
m
(
D
(
G
(
s
i
)
)
)
;
(
4
)
and
training the weight of the generator network G using a gradient descent algorithm;
(3.3) copying the weight of the generator network G to the network Ĝ to update the weight of the network G, every C times the training step (3.1) and (3.2) are performed; and
(4) completing the training of the discriminator network D and the generator network G after executing the step (3) for N train times,
wherein the radio resource manager is configured to input a current network state into the generator network G, the generator network G is configured to output the sampling values of the cumulative return distribution corresponding to each resource allocation strategy, a mean value of the sampling values of the return of each resource allocation strategy is calculated, and an action corresponding to a maximum mean value is taken as a resource allocation strategy of the radio resource manager; and
(5) allocating the wireless network resource according to the resource allocation strategy corresponding to the maximum mean value determined in step (4).
2. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein the discount factor γ is in the range of 0.75 to 0.9.
3. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein N is in the range of 30 to 55.
4. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein ϵ has an initial value of 0.9, and is reduced by 0.05 every 100 times the step (2) is performed and remains unchanged until ϵ reaches 0.05; and ξ is in the range of 0.8 to 1.5.
5. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein the magnitude N B of the buffer area is in the range of 3000 to 10000.
6. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein n d is in the range of 1 to 10; and the number m of the quadruplets is 32 or 64.
7. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein the penalty factor λ is 10, 20 or 30.
8. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein the gradient descent algorithm for training both the generator network G and the discriminator network D is Adam with a learning rate of 0.001.
9. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein K is in the range of 10 to 50.
10. The wireless network resource allocating method based on generative adversarial reinforcement learning according to claim 1 , wherein N train is in the range of 2000 to 3000.
11. A wireless network resource allocating device, comprising:
a processor; and
a memory for storing instructions executable by the processor,
wherein the processor is configured to execute the instructions in the memory to implement steps of the wireless network resource allocating method according to claim 1 .
12. The wireless network resource allocating device according to claim 11 , wherein the discount factor γ is in the range of 0.75 to 0.9.
13. The wireless network resource allocating device according to claim 11 , wherein N is in the range of 30 to 55.
14. The wireless network resource allocating device according to claim 11 , wherein ϵ has an initial value of 0.9, and is reduced by 0.05 every 100 times the step (2) is performed and remains unchanged until ϵ reaches 0.05; and (is in the range of 0.8 to 1.5.
15. The wireless network resource allocating device according to claim 11 , wherein the magnitude N B of the buffer area is in the range of 3000 to 10000; and the penalty factor λ is 10, 20 or 30.
16. The wireless network resource allocating device according to claim 11 , wherein n d is in the range of 1 to 10; and the number m of the quadruplets is 32 or 64.
17. The wireless network resource allocating device according to claim 11 , wherein the gradient descent algorithm for training both the generator network G and the discriminator network D is Adam with a learning rate of 0.001.
18. The wireless network resource allocating device according to claim 11 , wherein K is in the range of 10 to 50, and N train is in the range of 2000 to 3000.
19. A non-transitory computer-readable storage medium having stored therein executable instructions that, when executed by a processor, causes steps of the wireless network resource allocating method according to claim 1 to be implemented.
20. A wireless network resource allocating system, comprising:
a base station; and
a wireless network resource allocating device according to claim 11 , which is in communication with the base station,
wherein the base station is configured to transmit a plurality of network resource requests to the wireless network resource allocating device;
the wireless network resource allocating device is configured to receive the plurality of network resource requests from the base station, execute the wireless network resource allocating method according to the plurality of network resource requests to generate a resource allocation strategy, and transmit the resource allocation strategy to the base station; and
the base station is further configured to receive the resource allocation strategy from the wireless network resource allocating device, divide the network resource into a plurality of network slices according to the resource allocation strategy, and allocate the plurality of network slices.Cited by (0)
No later patents cite this yet.
References (0)
No backward citations on record.